Oct 31 01:03:24.143867 ip-10-0-140-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:03:24.549863 ip-10-0-140-164 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:24.549863 ip-10-0-140-164 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:03:24.549863 ip-10-0-140-164 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:24.549863 ip-10-0-140-164 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:03:24.555167 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555087    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.555167 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555168    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555172    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555176    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555180    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555184    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555187    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555190    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555192    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555195    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555197    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555200    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555202    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555205    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555208    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555210    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555213    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555215    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555218    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555226    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555229    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.555238 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555232    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555235    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555237    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555240    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555243    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555246    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555249    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555251    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555254    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555257    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555259    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555262    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555264    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555267    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555270    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555272    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555275    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555277    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555280    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.555710 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555282    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555284    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555287    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555289    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555292    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555294    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555297    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555299    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555304    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555308    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555311    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555315    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555322    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555325    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555327    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555330    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555333    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555335    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555338    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555341    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.556151 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555343    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555346    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555348    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555351    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555353    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555356    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555358    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555360    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555363    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555366    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555368    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555371    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555373    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555376    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555378    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555382    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555386    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555389    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555392    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.556630 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555395    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555397    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555400    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555402    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555406    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555409    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555412    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555415    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555418    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555421    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555423    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555426    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555428    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555431    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555433    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555437    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555440    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555442    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555445    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.557106 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555447    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555450    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555452    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555455    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555457    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555460    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555462    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.555465    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556283    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556290    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556293    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556296    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556299    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556302    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556305    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556307    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556310    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556313    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556316    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556318    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.557542 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556322    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556325    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556327    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556330    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556333    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556336    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556338    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556341    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556343    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556345    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556348    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556350    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556353    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556355    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556358    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556360    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556363    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556365    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556368    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556370    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.558082 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556372    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556375    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556378    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556380    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556383    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556385    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556388    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556390    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556393    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556396    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556399    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556401    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556404    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556407    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556410    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556413    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556415    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556418    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556420    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556423    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.558543 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556425    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556428    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556430    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556433    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556435    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556437    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556440    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556442    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556445    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556447    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556450    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556452    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556455    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556457    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556460    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556464    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556466    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556469    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556471    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.559095 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556474    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556476    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556479    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556482    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556484    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556487    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556490    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556493    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556495    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556498    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556500    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556503    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556505    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556508    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556511    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556513    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556516    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556518    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556521    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556523    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.559824 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556526    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556528    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556532    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556536    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556538    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556541    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556544    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556547    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556549    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556552    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556558    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556563    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556567    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556587    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.556590    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557072    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557076    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557079    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557082    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.560552 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557087    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557091    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557095    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557098    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557100    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557103    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557105    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557108    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557111    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557114    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557116    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557119    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557122    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557124    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557127    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557129    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557132    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557134    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557137    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557139    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.561388 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557142    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557144    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557147    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557152    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557155    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557158    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557160    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557163    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557166    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557169    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557172    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557174    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557177    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557180    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557183    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557185    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557188    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557190    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557193    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.562024 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557195    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557198    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557200    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557203    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557205    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557208    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557211    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557213    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557215    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557218    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557220    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557223    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557226    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557228    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557231    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557233    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557237    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557240    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557243    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557245    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.562597 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557248    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557251    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557253    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557256    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557258    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557261    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557263    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557266    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557269    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557272    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557274    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557277    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557279    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557282    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557285    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557287    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557289    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557292    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557294    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557297    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.563288 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557299    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557302    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557304    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557307    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557310    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557313    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557315    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557317    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557321    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557324    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557326    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557329    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557332    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557335    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557337    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557340    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557342    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557345    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557349    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.563937 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557353    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557356    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557359    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.557361    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558374    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558384    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558391    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558395    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558400    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558403    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558407    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558411    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558415    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558418    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558421    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558424    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558427    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558430    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558433    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558436    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558439    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558443    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558448    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558451    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:03:24.564469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558454    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558459    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558463    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558467    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558470    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558473    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558477    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558480    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558483    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558486    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558489    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558492    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558497    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558500    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558503    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558506    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558509    2574 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558512    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558517    2574 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558520    2574 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558522    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558525    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558529    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558533    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:03:24.565056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558536    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558539    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558542    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558545    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558548    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558551    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558555    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558558    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558561    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558565    2574 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558588    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558593    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558598    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558602    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558606    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558609    2574 flags.go:64] FLAG: --help="false"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558612    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558615    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558618    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558624    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558628    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558631    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558634    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:03:24.565691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558637    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558639    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558642    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558645    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558648    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558651    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558654    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558656    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558659    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558662    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558665    2574 flags.go:64] FLAG: --lock-file=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558668    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558671    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558674    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558679    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558682    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558685    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558688    2574 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558691    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558694    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558697    2574 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558700    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558704    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558707    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558711    2574 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:03:24.566256 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558714    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558717    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558720    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558723    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558726    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558729    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558732    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558740    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558743    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558746    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558749    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558752    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558755    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558758    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558761    2574 flags.go:64] FLAG: --port="10250"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558764    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558767    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0afe2e76924e24979"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558770    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558773    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558776    2574 flags.go:64] FLAG: --register-node="true"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558778    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558782    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558785    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558788    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558791    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558795    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:03:24.566864 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558798    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558801    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558804    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558807    2574 flags.go:64] FLAG: --runonce="false"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558809    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558812    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558815    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558818    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558821    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558824    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558827    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558830    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558834    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558837    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558840    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558842    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558845    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558848    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558851    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558856    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558859    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558862    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558867    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558870    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558872    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:03:24.568238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558875    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558878    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558883    2574 flags.go:64] FLAG: --v="2"
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558887    2574 flags.go:64] FLAG: --version="false"
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558892    2574 flags.go:64] FLAG: --vmodule=""
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558895    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.558899    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560350    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560366    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560371    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560376    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560380    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560385    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560389    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560394    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560398    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560403    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560409    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560413    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560417    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560421    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.568854 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560426    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560430    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560434    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560439    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560443    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560447    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560451    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560460    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560465    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560469    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560474    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560478    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560482    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560488    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560492    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560496    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560500    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560505    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560509    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560514    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.569345 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560519    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560523    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560528    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560532    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560536    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560541    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560545    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560549    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560553    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560557    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560561    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560565    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560590    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560596    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560600    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560604    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560608    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560612    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560617    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560621    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.569833 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560626    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560631    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560636    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560640    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560644    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560649    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560653    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560657    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560665    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560670    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560675    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560680    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560684    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560688    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560693    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560698    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560702    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560706    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560711    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.570339 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560715    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560719    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560723    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560728    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560732    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560737    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560741    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560746    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560750    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560754    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560768    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560772    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560777    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560782    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560788    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560794    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560799    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560804    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560811    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.570795 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560815    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560820    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560824    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560829    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560833    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560839    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560843    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560847    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560851    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560855    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560860    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560864    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560869    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560873    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.560921    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560980    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.571231 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560988    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560993    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.560997    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561001    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561006    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561010    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561014    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561018    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561034    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561039    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561043    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561047    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561051    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561055    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561060    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561064    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561071    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561075    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561079    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.571618 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561083    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561088    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561093    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561098    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561103    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561107    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561111    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561115    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561119    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561123    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561128    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561132    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561136    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561140    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561144    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561149    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561153    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561158    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561162    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561166    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.572060 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561171    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561176    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561188    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561194    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561198    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561202    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561206    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561210    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561215    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561221    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561225    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561229    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561234    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561238    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561244    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561248    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561253    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561257    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561261    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561266    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.572520 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561270    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561274    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561278    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561283    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561287    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561291    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561295    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561299    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561304    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561310    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561314    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561319    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561323    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561327    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561331    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561346    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561353    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561358    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561363    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.573074 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561367    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561371    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561377    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561381    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561385    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561389    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561394    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561399    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561403    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561407    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561411    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561415    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561420    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561424    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561429    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561433    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561437    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561442    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561446    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561450    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.573515 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561455    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561463    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561470    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561474    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561479    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561483    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.561488    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.569172    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.569188    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.569206    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.569211    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569267    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569272    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569275    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569277    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569280    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569283    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569285    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569288    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.573985 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569290    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569293    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569296    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569298    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569300    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569303    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569306    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569308    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569311    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569314    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569317    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569321    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569324    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569327    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569329    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569332    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569334    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569337    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569339    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569342    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.574425 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569344    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569349    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569352    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569355    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569358    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569361    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569363    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569366    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569368    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569371    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569374    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569376    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569379    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569381    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569384    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569386    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569389    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569391    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569394    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569396    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569399    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.574919 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569401    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569404    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569406    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569409    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569412    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569414    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569417    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569420    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569422    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569424    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569427    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569429    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569432    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569435    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569438    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569440    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569443    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569445    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569448    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.575429 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569451    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569453    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569455    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569458    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569461    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569463    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569465    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569468    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569470    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569473    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569475    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569478    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569480    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569483    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569485    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569489    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569493    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569496    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569499    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.575889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569502    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569505    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569508    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569510    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569513    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569515    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569518    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569521    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569524    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569526    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569529    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569531    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569534    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569536    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569538    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569541    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569543    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569546    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.576340 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569548    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.569591    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569731    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569735    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569738    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569741    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569743    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569746    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569749    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569751    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569754    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569756    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569759    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569763    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569766    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.576786 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569769    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569772    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569774    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569777    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569779    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569782    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569785    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569787    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569790    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569793    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569796    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569798    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569801    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569803    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569806    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569808    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569811    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569813    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569816    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569819    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569821    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.577132 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569824    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569827    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569829    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569831    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569834    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569837    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569840    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569842    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569845    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569848    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569850    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569853    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569855    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569858    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569861    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569863    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569865    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569868    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569871    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.577688 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569873    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569877    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569879    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569882    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569884    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569888    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569891    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569894    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569897    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569899    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569902    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569904    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569907    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569909    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569912    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569914    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569917    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569919    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569922    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.578142 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569924    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569927    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569929    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569932    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569934    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569937    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569939    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569942    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569945    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569948    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569950    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569955    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569958    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569960    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569963    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569966    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569969    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569971    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569974    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.578600 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569977    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569979    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569982    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569984    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569987    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569990    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569992    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569995    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.569997    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570000    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570003    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570005    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570008    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570010    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570020    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.570045    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:03:24.579037 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570078    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570083    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570086    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570090    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570094    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570097    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570099    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570102    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570105    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570109    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570112    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570114    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570117    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570120    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570123    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570125    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570128    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570130    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570133    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570136    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:03:24.579410 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570138    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570141    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570143    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570146    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570148    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570151    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570153    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570156    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570158    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570161    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570163    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570166    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570169    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570171    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570174    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570176    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570179    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570181    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570201    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570205    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:03:24.579889 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570207    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570211    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570214    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570216    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570219    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570222    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570225    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570227    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570230    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570232    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570238    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570241    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570244    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570246    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570249    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570252    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570254    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570257    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570259    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:03:24.580349 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570262    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570264    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570267    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570269    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570272    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570275    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570277    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570280    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570282    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570285    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570287    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570290    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570292    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570295    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570299    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570302    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570304    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570307    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570309    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570312    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:03:24.580951 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570315    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570318    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570321    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570323    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570326    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570328    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570331    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570333    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570336    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570338    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570341    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570344    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570347    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570350    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570352    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570355    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570359    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570361    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570364    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:03:24.581409 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570367    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570369    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570372    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570374    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570377    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570379    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570382    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:24.570385    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.570443    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.572826    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.573783    2574 server.go:1014] "Starting client certificate rotation"
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.573878    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.574632    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.575958    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:24.581925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.577238    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:03:24.590192 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.590173    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:03:24.594156 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.594142    2574 log.go:25] "Validated CRI v1 image API"
Oct 31 01:03:24.595017 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.594997    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:03:24.599437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.599415    2574 fs.go:135] Filesystem UUIDs: map[30bf5f50-d927-4b49-8fe4-926af04c3f16:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f50b57d3-a57c-48d5-b6ce-3c76c2fc40cb:/dev/nvme0n1p4]
Oct 31 01:03:24.599487 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.599438    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 01:03:24.605823 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.605707    2574 manager.go:217] Machine: {Timestamp:2026-10-31 01:03:24.604276306 +0000 UTC m=+0.347539069 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096664 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8dbf09d73ffedaff07d053a730ce SystemUUID:ec2e8dbf-09d7-3ffe-daff-07d053a730ce BootID:699e8e10-9adc-4ca2-ae32-cd9c7f960a48 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:32:be:3d:42:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:32:be:3d:42:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:91:66:26:ac:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:03:24.606465 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.606453    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:03:24.606549 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.606538    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:03:24.610075 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.610047    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:03:24.610206 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.610077    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-164.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:03:24.610915 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.610906    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:03:24.610945 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.610918    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:03:24.610979 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.610944    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:24.610979 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.610958    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:03:24.612261 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.612252    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:03:24.616881 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.616869    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:03:24.616997 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.616988    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:03:24.617780 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.617771    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:03:24.617808 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.617784    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:03:24.617808 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.617793    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:03:24.617808 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.617802    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:03:24.617887 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.617811    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:03:24.618151 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.618140    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:24.620511 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.620493    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:03:24.627854 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.627841    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:03:24.628775 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.628763    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:03:24.628809 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.628794    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:03:24.628975 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.628963    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:03:24.629008 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.628981    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:03:24.629008 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.628987    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:03:24.629008 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.628994    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:03:24.629080 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629010    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:03:24.629080 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629019    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:03:24.629649 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629641    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:03:24.629683 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629651    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:03:24.629683 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629658    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:03:24.629683 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629664    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:03:24.629683 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629677    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:03:24.629881 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.629686    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:03:24.631148 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.631137    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:03:24.631180 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.631152    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:03:24.635404 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.635391    2574 server.go:1263] "Started kubelet"
Oct 31 01:03:24.635525 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.635466    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:03:24.635625 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.635540    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:03:24.635625 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.635617    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:03:24.636115 ip-10-0-140-164 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:03:24.637234 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.637217    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:03:24.638226 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.638213    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:03:24.642123 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.642105    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 01:03:24.644868 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.644850    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:24.646154 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646139    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:03:24.646273 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.646253    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-164.us-west-1.compute.internal\" not found"
Oct 31 01:03:24.646887 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646872    2574 factory.go:55] Registering systemd factory
Oct 31 01:03:24.646887 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646884    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:03:24.647011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646891    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:03:24.647011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646889    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:03:24.647011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646909    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:03:24.647011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646978    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:03:24.647011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.646989    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:03:24.647214 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.647123    2574 factory.go:153] Registering CRI-O factory
Oct 31 01:03:24.647214 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.647136    2574 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:03:24.647214 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.647184    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:03:24.647214 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.647204    2574 factory.go:103] Registering Raw factory
Oct 31 01:03:24.647377 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.647217    2574 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:03:24.648814 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.648800    2574 manager.go:319] Starting recovery of all containers
Oct 31 01:03:24.660267 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.660250    2574 manager.go:324] Recovery completed
Oct 31 01:03:24.663921 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.663908    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.665281 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665260    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.665344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665302    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.665344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665318    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.665797 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665785    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:03:24.665848 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665796    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:03:24.665848 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665812    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:03:24.665848 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.665842    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.666467 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.666451    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.666521 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.666478    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.666521 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.666488    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.667906 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.667896    2574 policy_none.go:50] "Start"
Oct 31 01:03:24.667947 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.667911    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:03:24.667947 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.667923    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:03:24.669857 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.669846    2574 policy_none.go:44] "Start"
Oct 31 01:03:24.724240 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.724202    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:03:24.728132 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.728110    2574 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:03:24.728849 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.728830    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:03:24.728925 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.728856    2574 server.go:87] "Starting device plugin registration server"
Oct 31 01:03:24.729162 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.729151    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:03:24.729195 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.729166    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:03:24.729282 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.729268    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:03:24.729378 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.729368    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:03:24.729434 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.729379    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:03:24.730126 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.729965    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:03:24.730212 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.730161    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-164.us-west-1.compute.internal\" not found"
Oct 31 01:03:24.776233 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.776202    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-164.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 01:03:24.778136 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.776201    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-164.us-west-1.compute.internal.18e3779fc06b7c71  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-164.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-164.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:24.635364465 +0000 UTC m=+0.378627229,LastTimestamp:2026-10-31 01:03:24.635364465 +0000 UTC m=+0.378627229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-164.us-west-1.compute.internal,}"
Oct 31 01:03:24.800662 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.800593    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:03:24.801941 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.801920    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:03:24.801941 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.801944    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:03:24.802105 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.801967    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:03:24.802105 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.802007    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 01:03:24.824769 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.824745    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-164.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 01:03:24.830172 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.830157    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.831003 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.830985    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.831072 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.831024    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.831072 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.831035    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.831133 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.831078    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.858884 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.858863    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:03:24.858951 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.858940    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fpd7c"
Oct 31 01:03:24.902260 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.902231    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"]
Oct 31 01:03:24.902370 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.902298    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.903238 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.903212    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.903335 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.903247    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.903335 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.903262    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.905954 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.905938    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.906090 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.906075    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.906135 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.906105    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.906933 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.906913    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.907028 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.906945    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.907028 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.906959    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.907270 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.907229    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.907270 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.907250    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.907270 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.907264    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.909350 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.909332    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.909440 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.909363    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:03:24.910182 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.910159    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:03:24.910293 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.910192    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:03:24.910293 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.910206    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:03:24.934395 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.934376    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:24.935096 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.935081    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:24.943532 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.943514    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-164.us-west-1.compute.internal\" not found" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.945672 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:24.945657    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-164.us-west-1.compute.internal\" not found" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.947787 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.947772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.947832 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.947795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:24.947832 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:24.947811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7360d5bb713e28adcb32faa1b6003ade-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"7360d5bb713e28adcb32faa1b6003ade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.033690 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.033658    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:25.034330 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.034309    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.034408 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.034396    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:25.047349 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.047329    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.048338 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.048322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.048384 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.048348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.048384 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.048371    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7360d5bb713e28adcb32faa1b6003ade-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"7360d5bb713e28adcb32faa1b6003ade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.048448 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.048428    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.048448 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.048428    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.048507 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.048428    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7360d5bb713e28adcb32faa1b6003ade-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"7360d5bb713e28adcb32faa1b6003ade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.109676 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.109628    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:25.110202 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.110189    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.183587 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.183555    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:03:25.246097 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.246063    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.248309 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.248292    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:03:25.573780 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.573705    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 01:03:25.574318 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.573817    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 01:03:25.574318 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.573856    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 01:03:25.574318 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.573873    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 01:03:25.574318 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:25.573919    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a53752eb2738248268d838fbfc908485-8edbb764ce6e4113.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-164.us-west-1.compute.internal.18e3779fc2349bae\": read tcp 10.0.140.164:58284->3.151.241.160:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-164.us-west-1.compute.internal.18e3779fc2349bae  default   5160 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-164.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-164.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-164.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 01:03:24 +0000 UTC,LastTimestamp:2026-10-31 01:03:24.831039608 +0000 UTC m=+0.574302371,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-164.us-west-1.compute.internal,}"
Oct 31 01:03:25.618686 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.618660    2574 apiserver.go:52] "Watching apiserver"
Oct 31 01:03:25.645116 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.645093    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 01:03:25.863322 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.863237    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:58:26 +0000 UTC" deadline="2028-03-31 13:01:43.132337183 +0000 UTC"
Oct 31 01:03:25.863322 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.863267    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12419h58m17.269074728s"
Oct 31 01:03:25.880608 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.880564    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:03:25.880874 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.880855    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-qctlt","openshift-network-operator/iptables-alerter-kppxf","openshift-ovn-kubernetes/ovnkube-node-fpzqj","kube-system/konnectivity-agent-d9bhz","openshift-cluster-node-tuning-operator/tuned-wkfwh","openshift-multus/multus-additional-cni-plugins-b96bx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal","openshift-multus/multus-b5mnh","openshift-network-diagnostics/network-check-target-7znmf","kube-system/global-pull-secret-syncer-tmx2h","kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf","openshift-image-registry/node-ca-cmxkm","openshift-multus/network-metrics-daemon-vrg94"]
Oct 31 01:03:25.883288 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.883271    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:25.885960 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.885946    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:25.888085 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.888068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.890091 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.890077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:25.892173 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.892158    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.894290 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.894275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.896392 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.896378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.898530 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.898514    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:25.898625 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:25.898604    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:25.900874 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.900857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:25.900960 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:25.900938    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:25.903889 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.903869    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.906044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.906029    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:25.908268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.908250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:25.908324 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:25.908304    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:25.935951 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.935929    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g746\""
Oct 31 01:03:25.935951 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.935943    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.936064 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.935934    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:03:25.938618 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.938603    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.938703 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.938628    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:03:25.938703 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.938639    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42fft\""
Oct 31 01:03:25.938703 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.938674    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:25.940629 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940560    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:03:25.940629 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940596    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:03:25.940735 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940627    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:03:25.940735 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940676    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:03:25.940735 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940685    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.940831 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940805    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8rwd\""
Oct 31 01:03:25.940882 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.940874    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:03:25.942008 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.941994    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:03:25.942505 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.942493    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:03:25.942603 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.942592    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-55dd6\""
Oct 31 01:03:25.948652 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.948638    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:03:25.953985 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.953966    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkj5v\" (UniqueName: \"kubernetes.io/projected/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-kube-api-access-mkj5v\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.954068 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.953992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-netns\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954068 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954010    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-hostroot\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954068 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-etc-kubernetes\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954177 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-ovn\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.954177 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-log-socket\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.954177 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-sys-fs\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.954177 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-lib-modules\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.954344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954189    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9dd849-9bfc-44bf-8eba-b8a08677c824-konnectivity-ca\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:25.954344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-system-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-iptables-alerter-script\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:25.954344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-device-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.954344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de59f004-1680-492e-88f9-3278ed575e50-host\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:25.954344 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94xd\" (UniqueName: \"kubernetes.io/projected/549daf04-71a7-40cd-970b-0d14584c26a4-kube-api-access-x94xd\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954366    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-tmp-dir\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w99sz\" (UniqueName: \"kubernetes.io/projected/221a9c21-35c6-4628-87ca-0cc77c4b3595-kube-api-access-w99sz\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-kubelet\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954470    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954500    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-multus-certs\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954660 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954662    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-host-slash\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-netns\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954711    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-etc-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-run\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-multus\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-dbus\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954806    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954828    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mtd\" (UniqueName: \"kubernetes.io/projected/9cae5eb3-8cac-465d-9b07-f596954cf636-kube-api-access-96mtd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.954869 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954859    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfbg\" (UniqueName: \"kubernetes.io/projected/de59f004-1680-492e-88f9-3278ed575e50-kube-api-access-lkfbg\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-hosts-file\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954891    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-var-lib-kubelet\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-tmp\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954919    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-systemd-units\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954938    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-socket-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv56d\" (UniqueName: \"kubernetes.io/projected/8e7d6f01-4cbb-4865-931e-86cf217018fe-kube-api-access-jv56d\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.954988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-cni-binary-copy\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-kubelet-config\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4vd4\" (UniqueName: \"kubernetes.io/projected/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-kube-api-access-g4vd4\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-netd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955106    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-script-lib\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955131 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955123    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-cnibin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-multus-daemon-config\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-tuned\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9dd849-9bfc-44bf-8eba-b8a08677c824-agent-certs\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-modprobe-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-k8s-cni-cncf-io\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955330    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-node-log\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955362    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de59f004-1680-492e-88f9-3278ed575e50-serviceca\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955391    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-conf-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5v92\" (UniqueName: \"kubernetes.io/projected/120efa71-c477-4936-b599-e9f81f035a25-kube-api-access-b5v92\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-slash\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-systemd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-env-overrides\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955567    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-kubernetes\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.955721 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-socket-dir-parent\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-kubelet\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-bin\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cae5eb3-8cac-465d-9b07-f596954cf636-ovn-node-metrics-cert\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-sys\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955746    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cnibin\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955811    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w98f\" (UniqueName: \"kubernetes.io/projected/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-kube-api-access-9w98f\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysconfig\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955841    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-var-lib-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-conf\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955882    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-systemd\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-system-cni-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.956347 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.955990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-config\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:25.956873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.956033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:25.956873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.956063    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-bin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:25.956873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.956087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.956873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.956128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-registration-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:25.956873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.956165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-host\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:25.983437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.983415    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:03:25.983437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.983423    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9z86\""
Oct 31 01:03:25.983437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.983417    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.989792 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.989777    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kk499\""
Oct 31 01:03:25.989873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.989792    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:03:25.989873 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.989863    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zzhf8\""
Oct 31 01:03:25.989953 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.989884    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rnpg9\""
Oct 31 01:03:25.989953 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.989906    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-grw5x\""
Oct 31 01:03:25.994740 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.994724    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.994836 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.994823    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:03:25.995005 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.994992    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.995048 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995000    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:03:25.995180 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995164    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:03:25.995331 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995316    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:03:25.995366 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995329    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:03:25.995366 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995318    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:03:25.995366 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995362    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:03:25.995469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995369    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:03:25.995597 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:25.995586    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:03:26.034084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.034060    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 01:03:26.034203 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.034097    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tjtp8"
Oct 31 01:03:26.056941 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.056920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.057044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.056945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.056966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-config\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.056981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:26.057044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.056996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-bin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057014    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-registration-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057314 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057046    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057314 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057106    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-registration-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057314 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057111    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057314 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-bin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057314 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057236    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-host\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-host\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057384    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkj5v\" (UniqueName: \"kubernetes.io/projected/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-kube-api-access-mkj5v\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-netns\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-hostroot\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-etc-kubernetes\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-ovn\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-log-socket\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057497    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-sys-fs\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057493    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-netns\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057523    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-etc-kubernetes\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057536    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-log-socket\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057549    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-ovn\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057589    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-lib-modules\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057563    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-sys-fs\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9dd849-9bfc-44bf-8eba-b8a08677c824-konnectivity-ca\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057631    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-hostroot\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057635    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-config\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-system-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057734    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-iptables-alerter-script\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057741    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-lib-modules\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-device-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.057795 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057784    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-system-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de59f004-1680-492e-88f9-3278ed575e50-host\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057843    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-device-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94xd\" (UniqueName: \"kubernetes.io/projected/549daf04-71a7-40cd-970b-0d14584c26a4-kube-api-access-x94xd\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057908    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-tmp-dir\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.057933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99sz\" (UniqueName: \"kubernetes.io/projected/221a9c21-35c6-4628-87ca-0cc77c4b3595-kube-api-access-w99sz\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-kubelet\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058031    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de59f004-1680-492e-88f9-3278ed575e50-host\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058074    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9dd849-9bfc-44bf-8eba-b8a08677c824-konnectivity-ca\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058088    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-kubelet\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-multus-certs\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058119    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.058120    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-host-slash\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058215    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-host-slash\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.058450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058218    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-tmp-dir\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058225    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-iptables-alerter-script\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.058251    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:26.558231708 +0000 UTC m=+2.301494666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058253    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-multus-certs\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-netns\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058296    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058301    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-netns\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-etc-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058329    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-etc-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-run\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058353    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-run\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-multus\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-dbus\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058406    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-multus\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mtd\" (UniqueName: \"kubernetes.io/projected/9cae5eb3-8cac-465d-9b07-f596954cf636-kube-api-access-96mtd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfbg\" (UniqueName: \"kubernetes.io/projected/de59f004-1680-492e-88f9-3278ed575e50-kube-api-access-lkfbg\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.059304 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-hosts-file\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058460    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-dbus\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-var-lib-kubelet\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-tmp\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058507    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-hosts-file\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-systemd-units\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058526    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058530    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-var-lib-kubelet\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058563    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-systemd-units\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-socket-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv56d\" (UniqueName: \"kubernetes.io/projected/8e7d6f01-4cbb-4865-931e-86cf217018fe-kube-api-access-jv56d\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-cni-binary-copy\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-kubelet-config\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vd4\" (UniqueName: \"kubernetes.io/projected/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-kube-api-access-g4vd4\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-socket-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058747    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-netd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058781    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:03:26.060084 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-netd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058799    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-kubelet-config\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058858    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-script-lib\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-cnibin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-multus-daemon-config\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.058988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-tuned\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9dd849-9bfc-44bf-8eba-b8a08677c824-agent-certs\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059011    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-cnibin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059032    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-modprobe-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-k8s-cni-cncf-io\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-node-log\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de59f004-1680-492e-88f9-3278ed575e50-serviceca\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059108    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059140    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-conf-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059174    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v92\" (UniqueName: \"kubernetes.io/projected/120efa71-c477-4936-b599-e9f81f035a25-kube-api-access-b5v92\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059192    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-cni-binary-copy\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.060770 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059245    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059252    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-slash\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-systemd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059299    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-node-log\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059319    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-env-overrides\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-kubernetes\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059394    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-socket-dir-parent\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-kubelet\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-script-lib\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-bin\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059468    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-bin\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cae5eb3-8cac-465d-9b07-f596954cf636-ovn-node-metrics-cert\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059511    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-slash\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059555    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-systemd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059559    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-modprobe-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.061769 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-sys\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cnibin\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.059736    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.059773    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:26.559760786 +0000 UTC m=+2.303023536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w98f\" (UniqueName: \"kubernetes.io/projected/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-kube-api-access-9w98f\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059846    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysconfig\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-var-lib-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059872    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-conf-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059883    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059922    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-env-overrides\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059967    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-kubelet\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059961    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-socket-dir-parent\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059454    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-multus-daemon-config\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059994    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-conf\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.059886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-conf\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.062562 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060054    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-kubernetes\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060090    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cnibin\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060125    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-sys\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-systemd\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-system-cni-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060346    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-systemd\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060369    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysconfig\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060388    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-system-cni-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060431    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-var-lib-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.060594    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-k8s-cni-cncf-io\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.061337    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de59f004-1680-492e-88f9-3278ed575e50-serviceca\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.061596    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.062877    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-tmp\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.062915    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-tuned\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.063268 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.063061    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9dd849-9bfc-44bf-8eba-b8a08677c824-agent-certs\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:26.063688 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.063324    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cae5eb3-8cac-465d-9b07-f596954cf636-ovn-node-metrics-cert\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.121711 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.121645    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkj5v\" (UniqueName: \"kubernetes.io/projected/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-kube-api-access-mkj5v\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.123071 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.123053    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:26.123161 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.123074    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:26.123161 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.123086    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:26.123247 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:26.123162    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:26.623142427 +0000 UTC m=+2.366405191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:26.124104 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.124083    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfbg\" (UniqueName: \"kubernetes.io/projected/de59f004-1680-492e-88f9-3278ed575e50-kube-api-access-lkfbg\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.125366 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.125343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99sz\" (UniqueName: \"kubernetes.io/projected/221a9c21-35c6-4628-87ca-0cc77c4b3595-kube-api-access-w99sz\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.125469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.125342    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w98f\" (UniqueName: \"kubernetes.io/projected/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-kube-api-access-9w98f\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.126829 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.126808    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv56d\" (UniqueName: \"kubernetes.io/projected/8e7d6f01-4cbb-4865-931e-86cf217018fe-kube-api-access-jv56d\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.127469 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.127448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v92\" (UniqueName: \"kubernetes.io/projected/120efa71-c477-4936-b599-e9f81f035a25-kube-api-access-b5v92\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.127595 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.127472    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vd4\" (UniqueName: \"kubernetes.io/projected/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-kube-api-access-g4vd4\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.127957 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.127936    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mtd\" (UniqueName: \"kubernetes.io/projected/9cae5eb3-8cac-465d-9b07-f596954cf636-kube-api-access-96mtd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.128839 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.128823    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94xd\" (UniqueName: \"kubernetes.io/projected/549daf04-71a7-40cd-970b-0d14584c26a4-kube-api-access-x94xd\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:26.212680 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.212652    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:03:26.229921 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.229901    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:03:26.259212 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.259182    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:26.261568 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.261539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:26.273564 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.273539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:03:26.289435 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.289404    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b5mnh"
Oct 31 01:03:26.294085 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.294067    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:03:26.304853 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.304835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:03:26.307021 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.307004    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:03:26.367657 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:26.367614    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b44b15da499055b285bb46b1575da8d.slice/crio-7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589 WatchSource:0}: Error finding container 7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589: Status 404 returned error can't find the container with id 7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589
Oct 31 01:03:26.368008 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:26.367990    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7360d5bb713e28adcb32faa1b6003ade.slice/crio-5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af WatchSource:0}: Error finding container 5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af: Status 404 returned error can't find the container with id 5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af
Oct 31 01:03:26.373999 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:26.373972    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:03:27.848921 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:27.848896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:27.849053 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:27.848959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:27.849053 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.849043    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:27.849153 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.849050    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:27.849153 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.849115    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:28.849099575 +0000 UTC m=+3.306797299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:27.849153 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.849133    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:28.849123427 +0000 UTC m=+3.306821151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:27.949592 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:27.949511    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:27.950086 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.949656    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:27.950086 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.949674    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:27.950086 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.949685    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:27.950086 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:27.949732    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:28.9497175 +0000 UTC m=+3.407415224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:28.067159 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.067118    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:28.091743 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.091657    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerStarted","Data":"7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589"}
Oct 31 01:03:28.093006 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.092985    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"7360d5bb713e28adcb32faa1b6003ade","Type":"ContainerStarted","Data":"5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af"}
Oct 31 01:03:28.277982 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.277958    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5bbf79_871a_4f48_beea_cb3bf34fbe64.slice/crio-17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630 WatchSource:0}: Error finding container 17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630: Status 404 returned error can't find the container with id 17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630
Oct 31 01:03:28.287584 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.287553    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:28.322539 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.322498    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:27 +0000 UTC" deadline="2028-04-27 18:48:33.723637014 +0000 UTC"
Oct 31 01:03:28.322539 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.322537    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13073h45m5.401102903s"
Oct 31 01:03:28.331006 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.330986    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe8ed9e_59cf_4fc6_a568_aa8d71d82bda.slice/crio-0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a WatchSource:0}: Error finding container 0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a: Status 404 returned error can't find the container with id 0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a
Oct 31 01:03:28.339258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.339241    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:03:28.359878 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.359857    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9dd849_9bfc_44bf_8eba_b8a08677c824.slice/crio-ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b WatchSource:0}: Error finding container ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b: Status 404 returned error can't find the container with id ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b
Oct 31 01:03:28.371323 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.371297    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee4c566_ef1d_42ec_8e17_3de154e4ec69.slice/crio-b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00 WatchSource:0}: Error finding container b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00: Status 404 returned error can't find the container with id b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00
Oct 31 01:03:28.410333 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.410310    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7d6f01_4cbb_4865_931e_86cf217018fe.slice/crio-4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed WatchSource:0}: Error finding container 4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed: Status 404 returned error can't find the container with id 4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed
Oct 31 01:03:28.410987 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.410964    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221a9c21_35c6_4628_87ca_0cc77c4b3595.slice/crio-2a090addc726722a1ff96a0b6466e965f6c89b6bbfc66b9b5b9112c37e20bfec WatchSource:0}: Error finding container 2a090addc726722a1ff96a0b6466e965f6c89b6bbfc66b9b5b9112c37e20bfec: Status 404 returned error can't find the container with id 2a090addc726722a1ff96a0b6466e965f6c89b6bbfc66b9b5b9112c37e20bfec
Oct 31 01:03:28.428397 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.428378    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cae5eb3_8cac_465d_9b07_f596954cf636.slice/crio-ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0 WatchSource:0}: Error finding container ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0: Status 404 returned error can't find the container with id ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0
Oct 31 01:03:28.485330 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.485299    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120efa71_c477_4936_b599_e9f81f035a25.slice/crio-7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462 WatchSource:0}: Error finding container 7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462: Status 404 returned error can't find the container with id 7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462
Oct 31 01:03:28.507916 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:03:28.507892    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde59f004_1680_492e_88f9_3278ed575e50.slice/crio-f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2 WatchSource:0}: Error finding container f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2: Status 404 returned error can't find the container with id f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2
Oct 31 01:03:28.857432 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.856918    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:28.857432 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.856988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:28.857432 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.857190    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:28.857432 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.857295    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:30.857234455 +0000 UTC m=+5.314932182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:28.857432 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.857368    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:28.857432 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.857404    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:30.857393087 +0000 UTC m=+5.315090813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:28.957736 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:28.957690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:28.958439 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.957858    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:28.958439 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.957879    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:28.958439 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.957911    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:28.958439 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:28.957967    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:30.957949683 +0000 UTC m=+5.415647407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:29.088529 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.088495    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:29.088529 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.088520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:29.088529 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.088531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:29.088791 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:29.088644    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:29.088791 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:29.088734    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:29.088908 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:29.088849    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:29.096866 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.096838    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmxkm" event={"ID":"de59f004-1680-492e-88f9-3278ed575e50","Type":"ContainerStarted","Data":"f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2"}
Oct 31 01:03:29.099470 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.099443    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5mnh" event={"ID":"120efa71-c477-4936-b599-e9f81f035a25","Type":"ContainerStarted","Data":"7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462"}
Oct 31 01:03:29.100877 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.100853    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0"}
Oct 31 01:03:29.102222 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.102190    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wkfwh" event={"ID":"221a9c21-35c6-4628-87ca-0cc77c4b3595","Type":"ContainerStarted","Data":"2a090addc726722a1ff96a0b6466e965f6c89b6bbfc66b9b5b9112c37e20bfec"}
Oct 31 01:03:29.103482 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.103453    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerStarted","Data":"b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00"}
Oct 31 01:03:29.104781 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.104755    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qctlt" event={"ID":"aa5bbf79-871a-4f48-beea-cb3bf34fbe64","Type":"ContainerStarted","Data":"17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630"}
Oct 31 01:03:29.106090 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.106069    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf" event={"ID":"8e7d6f01-4cbb-4865-931e-86cf217018fe","Type":"ContainerStarted","Data":"4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed"}
Oct 31 01:03:29.107340 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.107318    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9bhz" event={"ID":"1c9dd849-9bfc-44bf-8eba-b8a08677c824","Type":"ContainerStarted","Data":"ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b"}
Oct 31 01:03:29.108527 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.108478    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kppxf" event={"ID":"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda","Type":"ContainerStarted","Data":"0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a"}
Oct 31 01:03:29.323389 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.323310    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:27 +0000 UTC" deadline="2028-06-21 23:42:17.513132326 +0000 UTC"
Oct 31 01:03:29.323389 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:29.323353    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14398h38m48.189784621s"
Oct 31 01:03:30.872984 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:30.872721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:30.872984 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:30.872804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:30.872984 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.872896    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:30.872984 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.872921    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:30.872984 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.872978    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:34.872955383 +0000 UTC m=+9.330653123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:30.873864 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.872998    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:34.872988585 +0000 UTC m=+9.330686309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:30.973572 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:30.973309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:30.973572 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.973471    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:30.973572 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.973487    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:30.973572 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.973495    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:30.973572 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:30.973539    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:34.973526873 +0000 UTC m=+9.431224610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:31.088342 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:31.087819    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:31.088342 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:31.088021    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:31.088342 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:31.088200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:31.088342 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:31.088201    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:31.088342 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:31.088307    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:31.088770 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:31.088360    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:31.158511 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:31.158424    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerStarted","Data":"84aad878c01028b90058c812ce1068a187282153e91e034ee1a8eca77e8ef9e5"}
Oct 31 01:03:32.169531 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:32.169268    2574 generic.go:356] "Generic (PLEG): container finished" podID="2b44b15da499055b285bb46b1575da8d" containerID="84aad878c01028b90058c812ce1068a187282153e91e034ee1a8eca77e8ef9e5" exitCode=0
Oct 31 01:03:32.169890 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:32.169566    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerDied","Data":"84aad878c01028b90058c812ce1068a187282153e91e034ee1a8eca77e8ef9e5"}
Oct 31 01:03:32.181531 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:32.181502    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="cf36fff776abb3dac0b3c7cfe58155471ff9889309936d362a18d479c410171d" exitCode=0
Oct 31 01:03:32.181713 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:32.181620    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerDied","Data":"cf36fff776abb3dac0b3c7cfe58155471ff9889309936d362a18d479c410171d"}
Oct 31 01:03:32.196437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:32.196409    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"7360d5bb713e28adcb32faa1b6003ade","Type":"ContainerStarted","Data":"d55e9063cb67474b8020c9ae7ac7f76195b6e7ef99fe055c20d9eaafaab64a74"}
Oct 31 01:03:33.088598 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.088530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:33.088724 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:33.088673    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:33.088758 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.088729    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:33.088817 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:33.088800    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:33.088856 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.088839    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:33.088893 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:33.088881    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:33.208931 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.208892    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerStarted","Data":"a2a92ccc907e89528c6d52ca5bb319a92afc4fcd7cb2ff3ce83ec3c0ed02dd5d"}
Oct 31 01:03:33.213086 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.213056    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-d9bhz" event={"ID":"1c9dd849-9bfc-44bf-8eba-b8a08677c824","Type":"ContainerStarted","Data":"f6c12e7d493db6a78ff561864a9ab98ee7abb7a3e2c6fc95e7d89fe0f54db277"}
Oct 31 01:03:33.327501 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.326297    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" podStartSLOduration=7.326279563 podStartE2EDuration="7.326279563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:33.325932895 +0000 UTC m=+7.783630643" watchObservedRunningTime="2026-10-31 01:03:33.326279563 +0000 UTC m=+7.783977307"
Oct 31 01:03:33.327501 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:33.326668    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" podStartSLOduration=7.326655005 podStartE2EDuration="7.326655005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:32.534878707 +0000 UTC m=+6.992576454" watchObservedRunningTime="2026-10-31 01:03:33.326655005 +0000 UTC m=+7.784352754"
Oct 31 01:03:34.093162 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:34.093022    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:34.093363 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:34.093132    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:34.703924 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:34.703893    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:34.704558 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:34.704397    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:34.818841 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:34.818787    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-d9bhz" podStartSLOduration=4.379802771 podStartE2EDuration="8.818772562s" totalImagesPullingTime="4.438969791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:33.440937307 +0000 UTC m=+7.898635052" watchObservedRunningTime="2026-10-31 01:03:34.818772562 +0000 UTC m=+9.276470320"
Oct 31 01:03:34.909155 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:34.909097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:34.909289 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:34.909172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:34.909289 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:34.909284    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:34.909400 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:34.909335    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:42.90931895 +0000 UTC m=+17.367016675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:34.909400 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:34.909379    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:34.909400 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:34.909401    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:42.909392828 +0000 UTC m=+17.367090552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:35.009511 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:35.009441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:35.009685 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:35.009616    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:35.009685 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:35.009635    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:35.009685 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:35.009647    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:35.009845 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:35.009695    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:43.009679553 +0000 UTC m=+17.467377277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:35.088671 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:35.088639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:35.088851 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:35.088790    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:35.088919 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:35.088864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:35.088979 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:35.088935    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:35.226079 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:35.225944    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf" event={"ID":"8e7d6f01-4cbb-4865-931e-86cf217018fe","Type":"ContainerStarted","Data":"5052b11508c933f6089daaa3c98179300c873c8b66b153ea9fc0676eab066a34"}
Oct 31 01:03:35.229536 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:35.229500    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cmxkm" event={"ID":"de59f004-1680-492e-88f9-3278ed575e50","Type":"ContainerStarted","Data":"b0b8d776fca3f563a219bcd7e19c92c7557f10a34a6042d885ca9fd3ce314093"}
Oct 31 01:03:35.353223 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:35.353163    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cmxkm" podStartSLOduration=3.872669225 podStartE2EDuration="9.353119275s" totalImagesPullingTime="5.48045005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:35.352865679 +0000 UTC m=+9.810563426" watchObservedRunningTime="2026-10-31 01:03:35.353119275 +0000 UTC m=+9.810817023"
Oct 31 01:03:36.092282 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:36.092250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:36.092776 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:36.092378    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:36.234590 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:36.233740    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:36.822495 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:36.822466    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:03:37.019710 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.019610    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:03:36.822491943Z","UUID":"d1349ce3-f731-4dc3-a727-fe900237b881","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:03:37.021299 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.021277    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:03:37.021398 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.021309    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:03:37.088216 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.088189    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:37.088331 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.088196    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:37.088331 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:37.088317    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:37.088404 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:37.088375    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:37.238359 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.238326    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerStarted","Data":"91b635864c295304f278f100131c3325e63e9f4dd083cd67efcc73f5ec6be8fa"}
Oct 31 01:03:37.239998 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.239969    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qctlt" event={"ID":"aa5bbf79-871a-4f48-beea-cb3bf34fbe64","Type":"ContainerStarted","Data":"edb20c4873604b23013f8dfe7b975d76984320cdedf1315ac2985ddc5ed021cd"}
Oct 31 01:03:37.242460 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.242434    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf" event={"ID":"8e7d6f01-4cbb-4865-931e-86cf217018fe","Type":"ContainerStarted","Data":"cd3b94d863d061b91a6e452cd6a4d63623507a3c63a56d1d4d2a3ccf2ee04a35"}
Oct 31 01:03:37.245738 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.245708    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b5mnh" event={"ID":"120efa71-c477-4936-b599-e9f81f035a25","Type":"ContainerStarted","Data":"f5c4a865bd19e6c3f882ec894a889a962e03a4a6fd12465eabc27fb32554ea79"}
Oct 31 01:03:37.479613 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.479499    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-b5mnh" podStartSLOduration=3.19846964 podStartE2EDuration="11.479481493s" totalImagesPullingTime="8.281011853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:37.47897378 +0000 UTC m=+11.936671556" watchObservedRunningTime="2026-10-31 01:03:37.479481493 +0000 UTC m=+11.937179241"
Oct 31 01:03:37.592810 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:37.592756    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-qctlt" podStartSLOduration=3.119864212 podStartE2EDuration="11.5927418s" totalImagesPullingTime="8.472877588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:37.592501419 +0000 UTC m=+12.050199165" watchObservedRunningTime="2026-10-31 01:03:37.5927418 +0000 UTC m=+12.050439546"
Oct 31 01:03:38.089292 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:38.089257    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:38.089453 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:38.089374    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:38.250521 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:38.250482    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="91b635864c295304f278f100131c3325e63e9f4dd083cd67efcc73f5ec6be8fa" exitCode=0
Oct 31 01:03:38.250913 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:38.250570    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerDied","Data":"91b635864c295304f278f100131c3325e63e9f4dd083cd67efcc73f5ec6be8fa"}
Oct 31 01:03:39.088467 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.088389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:39.088467 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.088434    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:39.088666 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:39.088513    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:39.088666 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:39.088614    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:39.254575 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.254535    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wkfwh" event={"ID":"221a9c21-35c6-4628-87ca-0cc77c4b3595","Type":"ContainerStarted","Data":"d4ba08101dd2ccbdab7c577244883b5d252d7be0ef4ad1c7f7745ee14e9c00ba"}
Oct 31 01:03:39.256995 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.256965    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf" event={"ID":"8e7d6f01-4cbb-4865-931e-86cf217018fe","Type":"ContainerStarted","Data":"7b5a8d5c84dffe915d0ffe569962dcb118968988a70f5d212d43e724402b8dfc"}
Oct 31 01:03:39.258536 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.258510    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kppxf" event={"ID":"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda","Type":"ContainerStarted","Data":"930c7cdc174e96b5f8dd523e5c494c23a5089edb504508afa8d62d63b83ae6ec"}
Oct 31 01:03:39.372299 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.372255    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wkfwh" podStartSLOduration=2.950056093 podStartE2EDuration="13.372241804s" totalImagesPullingTime="10.422185711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:39.371810204 +0000 UTC m=+13.829507950" watchObservedRunningTime="2026-10-31 01:03:39.372241804 +0000 UTC m=+13.829939549"
Oct 31 01:03:39.488908 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.488855    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kppxf" podStartSLOduration=4.823248953 podStartE2EDuration="13.488839345s" totalImagesPullingTime="8.665590392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:39.488363835 +0000 UTC m=+13.946061581" watchObservedRunningTime="2026-10-31 01:03:39.488839345 +0000 UTC m=+13.946537090"
Oct 31 01:03:39.604410 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:39.604323    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf" podStartSLOduration=3.51321659 podStartE2EDuration="13.604308291s" totalImagesPullingTime="10.091091701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:39.603976364 +0000 UTC m=+14.061674113" watchObservedRunningTime="2026-10-31 01:03:39.604308291 +0000 UTC m=+14.062006034"
Oct 31 01:03:40.088254 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:40.088179    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:40.088380 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:40.088291    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:40.263240 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:40.263201    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="13026f4aea9f89bc16fe7327d26c9b247ba427e2f02a9a4acb8238ea620213c4" exitCode=0
Oct 31 01:03:40.263941 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:40.263308    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerDied","Data":"13026f4aea9f89bc16fe7327d26c9b247ba427e2f02a9a4acb8238ea620213c4"}
Oct 31 01:03:41.088697 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:41.088661    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:41.088697 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:41.088707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:41.088933 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:41.088805    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:41.088983 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:41.088926    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:42.091803 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:42.091776    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:42.092434 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:42.091896    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:42.270082 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:42.270045    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="241b20c97e7aaff3768c61fa2dce7cf396edbc90b1b9c18f9cce9b4b0afd41de" exitCode=0
Oct 31 01:03:42.270269 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:42.270105    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerDied","Data":"241b20c97e7aaff3768c61fa2dce7cf396edbc90b1b9c18f9cce9b4b0afd41de"}
Oct 31 01:03:42.978772 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:42.978744    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:42.978958 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:42.978788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:42.978958 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:42.978894    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:42.978958 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:42.978894    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:42.978958 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:42.978951    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:58.978938495 +0000 UTC m=+33.436636222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:42.979176 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:42.978965    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:58.978958844 +0000 UTC m=+33.436656568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:43.080017 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:43.079980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:43.080193 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:43.080164    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:43.080193 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:43.080187    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:43.080283 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:43.080199    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:43.080283 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:43.080258    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:03:59.080239895 +0000 UTC m=+33.537937635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:43.088059 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:43.088031    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:43.088211 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:43.088072    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:43.088211 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:43.088194    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:43.088284 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:43.088271    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:44.092861 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.092625    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:44.093337 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:44.092965    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:44.278209 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.278180    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"ffead0d121988ec3ed69f4ce098bd5df4ded63748874e0a73b9b75e268e6dd94"}
Oct 31 01:03:44.278376 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.278255    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"80cf48259d468bc1bf1fe396e0f06b828550b110829fde44516c58377769c1f3"}
Oct 31 01:03:44.278376 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.278307    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"6ab1c659ea5e5f8e994a71e97ec30e5378e466d77f1a2cd93c9f63f4b57d2d8f"}
Oct 31 01:03:44.278376 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.278350    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"2c53d3f369a9815ba12df0bad20006af0182386ab757588b2db9a85d44f3d484"}
Oct 31 01:03:44.278556 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.278388    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"b006685e6fc71632a512fc9e2ffa210e138774d778b5c0a6e1b9c322913a36c8"}
Oct 31 01:03:44.278556 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:44.278432    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"08da42f2d1ebd4d1babc36e9f257d08677f3d382cf844434267c22eff9259634"}
Oct 31 01:03:45.088001 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:45.087921    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:45.088001 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:45.087962    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:45.088201 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:45.088043    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:45.088201 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:45.088056    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:45.283011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:45.282980    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="5be993e651766de9b9cffda59ca42f29d7c7235884f4bb8598345e4d1090650a" exitCode=0
Oct 31 01:03:45.283409 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:45.283027    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerDied","Data":"5be993e651766de9b9cffda59ca42f29d7c7235884f4bb8598345e4d1090650a"}
Oct 31 01:03:46.088883 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:46.088851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:46.089034 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:46.088939    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:46.287494 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:46.287421    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"afaa0d20ddadb3d1e0ef86e4118bd9907af970351f7052ed2f5510dda7c4c400"}
Oct 31 01:03:46.290398 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:46.290362    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="b86e389df5bc3e46369bf54a37f6867f022ceee764341582da83bc246ca4be45" exitCode=0
Oct 31 01:03:46.290509 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:46.290402    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerDied","Data":"b86e389df5bc3e46369bf54a37f6867f022ceee764341582da83bc246ca4be45"}
Oct 31 01:03:47.088394 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:47.088369    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:47.088566 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:47.088473    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:47.088566 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:47.088498    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:47.088566 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:47.088540    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:47.295159 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:47.295117    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b96bx" event={"ID":"6ee4c566-ef1d-42ec-8e17-3de154e4ec69","Type":"ContainerStarted","Data":"4c6cc01ea3cf6b75db1f99027e272df7629e2163a24920729dcb10848a588e57"}
Oct 31 01:03:47.451027 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:47.450943    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b96bx" podStartSLOduration=9.126126963 podStartE2EDuration="21.450929949s" totalImagesPullingTime="12.324802986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:47.449893975 +0000 UTC m=+21.907591721" watchObservedRunningTime="2026-10-31 01:03:47.450929949 +0000 UTC m=+21.908627695"
Oct 31 01:03:48.094391 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:48.094362    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:48.094532 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:48.094508    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:48.301524 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:48.301434    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" event={"ID":"9cae5eb3-8cac-465d-9b07-f596954cf636","Type":"ContainerStarted","Data":"7439256cc0be0aa764da47d64bb9c0bc822a9c57a4a6f893e9d95355b1059f39"}
Oct 31 01:03:48.530243 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:48.530197    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj" podStartSLOduration=7.707469879 podStartE2EDuration="22.530184257s" totalImagesPullingTime="14.822714378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:48.479733288 +0000 UTC m=+22.937431033" watchObservedRunningTime="2026-10-31 01:03:48.530184257 +0000 UTC m=+22.987882004"
Oct 31 01:03:49.088479 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.088444    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:49.088725 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:49.088552    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:49.088725 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.088616    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:49.088840 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:49.088722    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:49.304218 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.304179    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:49.304645 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.304258    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:49.304645 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.304306    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:49.323297 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.323270    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:49.325906 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:49.325877    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:03:50.088360 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.088328    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:50.088508 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:50.088437    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:50.924318 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.924236    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tmx2h"]
Oct 31 01:03:50.924688 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.924356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:50.924688 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:50.924446    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:50.929259 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.929234    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrg94"]
Oct 31 01:03:50.929537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.929518    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:50.929665 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:50.929613    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:50.974252 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.974226    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7znmf"]
Oct 31 01:03:50.974342 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:50.974313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:50.974394 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:50.974374    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:51.838938 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:51.838902    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d9bhz_1c9dd849-9bfc-44bf-8eba-b8a08677c824/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:52.091421 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:52.091355    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:52.091779 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:52.091471    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:52.091779 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:52.091518    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:52.091779 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:52.091612    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:52.243094 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:52.243054    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d9bhz_1c9dd849-9bfc-44bf-8eba-b8a08677c824/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:52.569880 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:52.569807    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d9bhz_1c9dd849-9bfc-44bf-8eba-b8a08677c824/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:52.841058 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:52.840966    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-d9bhz_1c9dd849-9bfc-44bf-8eba-b8a08677c824/konnectivity-agent/0.log" limit=1024
Oct 31 01:03:52.893760 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:52.893724    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qctlt_aa5bbf79-871a-4f48-beea-cb3bf34fbe64/dns-node-resolver/0.log"
Oct 31 01:03:53.088599 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:53.088570    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:53.088755 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:53.088664    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:53.407151 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:53.407109    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmxkm_de59f004-1680-492e-88f9-3278ed575e50/node-ca/0.log"
Oct 31 01:03:54.087778 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:54.087748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:54.087947 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:54.087750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:54.087947 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:54.087849    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:54.088030 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:54.087949    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:55.088237 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:55.088209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:55.088706 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:55.088338    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:56.090536 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:56.090504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:56.090883 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:56.090621    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:56.090883 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:56.090678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:56.090883 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:56.090764    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:56.168218 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:56.168181    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:56.168372 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:56.168324    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:03:56.168852 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:56.168836    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:03:57.088278 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:57.088248    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:57.088426 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:57.088347    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:58.088340 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:58.088311    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:58.088746 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:58.088432    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:03:58.088746 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:58.088517    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:58.088746 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:58.088604    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:03:58.991661 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:58.991621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:03:58.991825 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:58.991674    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:03:58.991825 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:58.991767    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:58.991825 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:58.991774    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:58.991825 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:58.991819    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret podName:ef840aa5-db4c-4422-a2d4-702a5c8bdcb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:30.991805516 +0000 UTC m=+65.449503241 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret") pod "global-pull-secret-syncer-tmx2h" (UID: "ef840aa5-db4c-4422-a2d4-702a5c8bdcb0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 01:03:58.991952 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:58.991832    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs podName:549daf04-71a7-40cd-970b-0d14584c26a4 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:30.991825088 +0000 UTC m=+65.449522813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs") pod "network-metrics-daemon-vrg94" (UID: "549daf04-71a7-40cd-970b-0d14584c26a4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 01:03:59.087903 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:59.087871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:59.088048 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:59.087968    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:03:59.092068 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:03:59.092048    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:03:59.092366 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:59.092208    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 01:03:59.092366 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:59.092226    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 01:03:59.092366 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:59.092236    2574 projected.go:196] Error preparing data for projected volume kube-api-access-nbt4j for pod openshift-network-diagnostics/network-check-target-7znmf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:03:59.092366 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:03:59.092278    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j podName:2fa45a8e-78a1-4cce-96e8-5ba324073c61 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:31.092265888 +0000 UTC m=+65.549963612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbt4j" (UniqueName: "kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j") pod "network-check-target-7znmf" (UID: "2fa45a8e-78a1-4cce-96e8-5ba324073c61") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 01:04:00.088631 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:00.088597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:00.088852 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:00.088699    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:00.088852 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:00.088742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:00.088852 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:00.088800    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:01.087942 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:01.087913    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:01.088335 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:01.088003    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:04:02.088403 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:02.088368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:02.088832 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:02.088506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:02.088832 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:02.088504    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:02.088832 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:02.088611    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:03.087711 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:03.087676    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:03.087896 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:03.087796    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:04:04.087813 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:04.087778    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:04.088295 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:04.087826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:04.088295 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:04.087912    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:04.088295 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:04.088021    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:05.088465 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:05.088430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:05.088900 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:05.088553    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:04:06.089231 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:06.089198    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:06.089594 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:06.089280    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:06.089594 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:06.089313    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:06.089594 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:06.089408    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:07.087990 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:07.087954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:07.088162 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:07.088054    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:04:08.088696 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:08.088666    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:08.089031 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:08.088777    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:08.089031 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:08.088850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:08.089031 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:08.088942    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:09.088537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:09.088505    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:09.088706 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:09.088600    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:04:10.088002 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:10.087961    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:10.088002 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:10.087985    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:10.088507 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:10.088093    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:10.088507 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:10.088251    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:11.088027 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:11.087996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:11.088423 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:11.088096    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7znmf" podUID="2fa45a8e-78a1-4cce-96e8-5ba324073c61"
Oct 31 01:04:12.088215 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:12.088181    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:12.088681 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:12.088184    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:12.088681 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:12.088294    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vrg94" podUID="549daf04-71a7-40cd-970b-0d14584c26a4"
Oct 31 01:04:12.088681 ip-10-0-140-164 kubenswrapper[2574]: E1031 01:04:12.088397    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tmx2h" podUID="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0"
Oct 31 01:04:12.829816 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:12.829786    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeReady"
Oct 31 01:04:12.829994 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:12.829913    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 01:04:12.985107 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:12.985077    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-btnvg"]
Oct 31 01:04:13.038831 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.038798    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dnvvs"]
Oct 31 01:04:13.038975 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.038928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.051909 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.051888    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-btnvg"]
Oct 31 01:04:13.052015 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.051925    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnvvs"]
Oct 31 01:04:13.052015 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.051940    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zx8f4"]
Oct 31 01:04:13.052015 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.051990    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.066212 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.066191    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.087905 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.087861    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:13.093369 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.093345    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:04:13.093742 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.093450    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:04:13.093742 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.093483    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sjdn\""
Oct 31 01:04:13.093742 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.093526    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:04:13.094112 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.094079    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:04:13.094839 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.094822    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zx8f4"]
Oct 31 01:04:13.106011 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.105994    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:04:13.106281 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.106253    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:04:13.106281 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.106274    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xgzbp\""
Oct 31 01:04:13.106717 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.106704    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:04:13.189970 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.189945    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96abcc08-0354-41d9-baec-af53dd804fdd-metrics-tls\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.190104 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.189978    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0e6596-aec8-43f1-9aac-2497c9563bef-data-volume\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.190104 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.189998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cfp\" (UniqueName: \"kubernetes.io/projected/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-api-access-78cfp\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.190203 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96abcc08-0354-41d9-baec-af53dd804fdd-config-volume\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.190203 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0e6596-aec8-43f1-9aac-2497c9563bef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.190264 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/198bdfe6-2b3f-4426-a0cc-6568d22b139d-cert\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.190264 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8d4p\" (UniqueName: \"kubernetes.io/projected/96abcc08-0354-41d9-baec-af53dd804fdd-kube-api-access-t8d4p\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.190264 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190253    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtv7v\" (UniqueName: \"kubernetes.io/projected/198bdfe6-2b3f-4426-a0cc-6568d22b139d-kube-api-access-qtv7v\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.190348 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.190348 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0e6596-aec8-43f1-9aac-2497c9563bef-crio-socket\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.190410 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.190363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96abcc08-0354-41d9-baec-af53dd804fdd-tmp-dir\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.193480 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.193464    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:04:13.193684 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.193660    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:04:13.193786 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.193714    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:04:13.193828 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.193805    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:04:13.194132 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.194121    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvb7t\""
Oct 31 01:04:13.194207 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.194130    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:04:13.290599 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96abcc08-0354-41d9-baec-af53dd804fdd-config-volume\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.290756 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0e6596-aec8-43f1-9aac-2497c9563bef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.290756 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/198bdfe6-2b3f-4426-a0cc-6568d22b139d-cert\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.290756 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290646    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8d4p\" (UniqueName: \"kubernetes.io/projected/96abcc08-0354-41d9-baec-af53dd804fdd-kube-api-access-t8d4p\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.290884 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290801    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtv7v\" (UniqueName: \"kubernetes.io/projected/198bdfe6-2b3f-4426-a0cc-6568d22b139d-kube-api-access-qtv7v\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.290884 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.290884 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0e6596-aec8-43f1-9aac-2497c9563bef-crio-socket\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.291024 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96abcc08-0354-41d9-baec-af53dd804fdd-tmp-dir\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.291024 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290943    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96abcc08-0354-41d9-baec-af53dd804fdd-metrics-tls\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.291024 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0e6596-aec8-43f1-9aac-2497c9563bef-data-volume\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.291024 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.290996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cfp\" (UniqueName: \"kubernetes.io/projected/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-api-access-78cfp\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.291249 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.291116    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0e6596-aec8-43f1-9aac-2497c9563bef-crio-socket\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.291429 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.291409    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96abcc08-0354-41d9-baec-af53dd804fdd-config-volume\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.291501 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.291433    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.291501 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.291462    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96abcc08-0354-41d9-baec-af53dd804fdd-tmp-dir\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.291626 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.291560    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0e6596-aec8-43f1-9aac-2497c9563bef-data-volume\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.294400 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.294378    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0e6596-aec8-43f1-9aac-2497c9563bef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.294491 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.294415    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96abcc08-0354-41d9-baec-af53dd804fdd-metrics-tls\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.294491 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.294446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/198bdfe6-2b3f-4426-a0cc-6568d22b139d-cert\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.347686 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.347628    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8d4p\" (UniqueName: \"kubernetes.io/projected/96abcc08-0354-41d9-baec-af53dd804fdd-kube-api-access-t8d4p\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.347779 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.347694    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cfp\" (UniqueName: \"kubernetes.io/projected/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-api-access-78cfp\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.348520 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.348502    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtv7v\" (UniqueName: \"kubernetes.io/projected/198bdfe6-2b3f-4426-a0cc-6568d22b139d-kube-api-access-qtv7v\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.386955 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.386931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:13.408581 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.408557    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:13.457157 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.457105    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:13.898796 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.898629    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnvvs"]
Oct 31 01:04:13.899329 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.899301    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-btnvg"]
Oct 31 01:04:13.900810 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:13.900790    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zx8f4"]
Oct 31 01:04:13.903128 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:13.903099    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198bdfe6_2b3f_4426_a0cc_6568d22b139d.slice/crio-f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607 WatchSource:0}: Error finding container f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607: Status 404 returned error can't find the container with id f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607
Oct 31 01:04:13.903687 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:13.903669    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0e6596_aec8_43f1_9aac_2497c9563bef.slice/crio-d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87 WatchSource:0}: Error finding container d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87: Status 404 returned error can't find the container with id d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87
Oct 31 01:04:13.940598 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:13.940576    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96abcc08_0354_41d9_baec_af53dd804fdd.slice/crio-fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35 WatchSource:0}: Error finding container fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35: Status 404 returned error can't find the container with id fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35
Oct 31 01:04:14.087783 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.087747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:14.087975 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.087757    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:14.140883 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.140854    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:04:14.143800 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.143769    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:04:14.143943 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.143769    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:04:14.143943 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.143861    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:04:14.355279 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.355184    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zx8f4" event={"ID":"96abcc08-0354-41d9-baec-af53dd804fdd","Type":"ContainerStarted","Data":"fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35"}
Oct 31 01:04:14.356392 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.356362    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnvvs" event={"ID":"198bdfe6-2b3f-4426-a0cc-6568d22b139d","Type":"ContainerStarted","Data":"f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607"}
Oct 31 01:04:14.358269 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.358247    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btnvg" event={"ID":"af0e6596-aec8-43f1-9aac-2497c9563bef","Type":"ContainerStarted","Data":"0b812b168c0736398096f7e7846c03e89223c2b7816c96fc5b0ec0845fb6cf1b"}
Oct 31 01:04:14.358350 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.358290    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btnvg" event={"ID":"af0e6596-aec8-43f1-9aac-2497c9563bef","Type":"ContainerStarted","Data":"d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87"}
Oct 31 01:04:14.919094 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.919058    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"]
Oct 31 01:04:14.932675 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.932646    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:14.980759 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:14.980725    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"]
Oct 31 01:04:15.029998 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.029959    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 01:04:15.029998 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.029997    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:04:15.035252 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.035226    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cm644\""
Oct 31 01:04:15.035252 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.035232    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:04:15.035417 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.035230    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 01:04:15.035417 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.035351    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:04:15.103680 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.103646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.103862 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.103715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71865382-46e6-4f6a-bb1f-9b5406439186-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.103862 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.103841    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.103969 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.103900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzj7\" (UniqueName: \"kubernetes.io/projected/71865382-46e6-4f6a-bb1f-9b5406439186-kube-api-access-9nzj7\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.204470 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.204390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.204470 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.204464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71865382-46e6-4f6a-bb1f-9b5406439186-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.204919 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.204507    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.204919 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.204538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzj7\" (UniqueName: \"kubernetes.io/projected/71865382-46e6-4f6a-bb1f-9b5406439186-kube-api-access-9nzj7\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.205742 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.205719    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71865382-46e6-4f6a-bb1f-9b5406439186-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.208482 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.208457    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.217626 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.217598    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.263691 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.262927    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzj7\" (UniqueName: \"kubernetes.io/projected/71865382-46e6-4f6a-bb1f-9b5406439186-kube-api-access-9nzj7\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.266384 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.266333    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:15.669249 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:15.669224    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"]
Oct 31 01:04:15.673065 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:15.673013    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71865382_46e6_4f6a_bb1f_9b5406439186.slice/crio-e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6 WatchSource:0}: Error finding container e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6: Status 404 returned error can't find the container with id e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6
Oct 31 01:04:16.365110 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:16.365077    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btnvg" event={"ID":"af0e6596-aec8-43f1-9aac-2497c9563bef","Type":"ContainerStarted","Data":"34d141e931835498886eeccddc832ef018c2643d14b1cd3c791bb70d7b1bdf52"}
Oct 31 01:04:16.366437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:16.366407    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz" event={"ID":"71865382-46e6-4f6a-bb1f-9b5406439186","Type":"ContainerStarted","Data":"e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6"}
Oct 31 01:04:17.371150 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.371100    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zx8f4" event={"ID":"96abcc08-0354-41d9-baec-af53dd804fdd","Type":"ContainerStarted","Data":"de6d209c848e6b67a2732c7330661305cc26772769c3b29bb22c907730f6bb4f"}
Oct 31 01:04:17.371649 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.371225    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zx8f4" event={"ID":"96abcc08-0354-41d9-baec-af53dd804fdd","Type":"ContainerStarted","Data":"b3fbde4a06978105a0ea55a32dafc929fe554b70344abdd4ecfd4d7c9d4f8c4e"}
Oct 31 01:04:17.371649 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.371258    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:17.492200 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.491955    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zx8f4" podStartSLOduration=2.7695454870000003 podStartE2EDuration="5.491937117s" totalImagesPullingTime="2.72239163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:17.491683715 +0000 UTC m=+51.949381472" watchObservedRunningTime="2026-10-31 01:04:17.491937117 +0000 UTC m=+51.949634863"
Oct 31 01:04:17.859409 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.859375    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-9p7q6"]
Oct 31 01:04:17.883474 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.883445    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:17.920455 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.920428    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9p7q6"]
Oct 31 01:04:17.936634 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.936606    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 01:04:17.936634 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.936606    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 01:04:17.936942 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:17.936924    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6cbrt\""
Oct 31 01:04:18.027120 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.027086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/111a48d2-3e5c-4c68-8215-98b0cfc5b301-tmp\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.027261 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.027223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngvl\" (UniqueName: \"kubernetes.io/projected/111a48d2-3e5c-4c68-8215-98b0cfc5b301-kube-api-access-tngvl\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.128044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.127982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngvl\" (UniqueName: \"kubernetes.io/projected/111a48d2-3e5c-4c68-8215-98b0cfc5b301-kube-api-access-tngvl\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.128044 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.128018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/111a48d2-3e5c-4c68-8215-98b0cfc5b301-tmp\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.128366 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.128351    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/111a48d2-3e5c-4c68-8215-98b0cfc5b301-tmp\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.196404 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.196377    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngvl\" (UniqueName: \"kubernetes.io/projected/111a48d2-3e5c-4c68-8215-98b0cfc5b301-kube-api-access-tngvl\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.214450 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.214430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:18.921900 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:18.921871    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9p7q6"]
Oct 31 01:04:18.962804 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:18.962777    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111a48d2_3e5c_4c68_8215_98b0cfc5b301.slice/crio-ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458 WatchSource:0}: Error finding container ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458: Status 404 returned error can't find the container with id ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458
Oct 31 01:04:19.379867 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.379830    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9p7q6" event={"ID":"111a48d2-3e5c-4c68-8215-98b0cfc5b301","Type":"ContainerStarted","Data":"ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458"}
Oct 31 01:04:19.381906 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.381881    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-btnvg" event={"ID":"af0e6596-aec8-43f1-9aac-2497c9563bef","Type":"ContainerStarted","Data":"8f303bf94ee69295d2b9c04de6a9fc13358d92260aae0e421215e525bb95dbca"}
Oct 31 01:04:19.384001 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.383979    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz" event={"ID":"71865382-46e6-4f6a-bb1f-9b5406439186","Type":"ContainerStarted","Data":"70a3852dcd0dea8dd2109adf77009e65e5ef1d1fffeedd05dc67cb5ec7b1dd20"}
Oct 31 01:04:19.384125 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.384028    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz" event={"ID":"71865382-46e6-4f6a-bb1f-9b5406439186","Type":"ContainerStarted","Data":"79d915a2ffc249c99136701ef52e9ac9ca6c5c646268c4c05b1867f5f9330f7f"}
Oct 31 01:04:19.385422 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.385406    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnvvs" event={"ID":"198bdfe6-2b3f-4426-a0cc-6568d22b139d","Type":"ContainerStarted","Data":"1eddd336f3bf3e897bf69e57689b4e96c190ea03e897ff0a7b738ce9cd76a079"}
Oct 31 01:04:19.502475 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.502424    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-btnvg" podStartSLOduration=3.143993503 podStartE2EDuration="7.502409363s" totalImagesPullingTime="4.35841586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:04:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:19.501639739 +0000 UTC m=+53.959337484" watchObservedRunningTime="2026-10-31 01:04:19.502409363 +0000 UTC m=+53.960107148"
Oct 31 01:04:19.622088 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.622035    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dnvvs" podStartSLOduration=3.068598469 podStartE2EDuration="7.622015791s" totalImagesPullingTime="4.553417322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:19.621854458 +0000 UTC m=+54.079552205" watchObservedRunningTime="2026-10-31 01:04:19.622015791 +0000 UTC m=+54.079713538"
Oct 31 01:04:19.740285 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:19.740230    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz" podStartSLOduration=2.957227778 podStartE2EDuration="5.740214088s" totalImagesPullingTime="2.78298631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:19.739166939 +0000 UTC m=+54.196864700" watchObservedRunningTime="2026-10-31 01:04:19.740214088 +0000 UTC m=+54.197911828"
Oct 31 01:04:21.321157 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:21.321105    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:22.698345 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.698283    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fbjh9"]
Oct 31 01:04:22.700421 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.700397    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.756395 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.756370    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:04:22.806777 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.806745    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ln2ts\""
Oct 31 01:04:22.806777 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.806775    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:04:22.806984 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.806809    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:04:22.867579 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867538    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-tls\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.867736 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-metrics-client-ca\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.867736 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867612    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-sys\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.867736 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-wtmp\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.867918 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-root\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.867918 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.867918 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65jc4\" (UniqueName: \"kubernetes.io/projected/43cb7dd9-ef71-43ac-8631-84d0ee80468a-kube-api-access-65jc4\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.868061 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.867950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-textfile\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.868061 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.868002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969022 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.968982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-tls\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969022 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-metrics-client-ca\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-sys\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969080    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-wtmp\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-root\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969171    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-sys\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969180    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969240    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-wtmp\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969258 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65jc4\" (UniqueName: \"kubernetes.io/projected/43cb7dd9-ef71-43ac-8631-84d0ee80468a-kube-api-access-65jc4\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969498 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-textfile\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969498 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969307    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-root\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969498 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.969816 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969759    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-metrics-client-ca\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.970005 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.969953    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.970079 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.970060    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-textfile\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.971700 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.971673    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-tls\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:22.972069 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:22.972052    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:23.026847 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:23.026816    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65jc4\" (UniqueName: \"kubernetes.io/projected/43cb7dd9-ef71-43ac-8631-84d0ee80468a-kube-api-access-65jc4\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:23.314486 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:23.314397    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:23.324962 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:23.324931    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb7dd9_ef71_43ac_8631_84d0ee80468a.slice/crio-6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749 WatchSource:0}: Error finding container 6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749: Status 404 returned error can't find the container with id 6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749
Oct 31 01:04:23.398224 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:23.398181    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbjh9" event={"ID":"43cb7dd9-ef71-43ac-8631-84d0ee80468a","Type":"ContainerStarted","Data":"6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749"}
Oct 31 01:04:25.017632 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.017558    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:04:25.019622 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.019605    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.085819 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.085783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.085819 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.085827    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086230 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.085928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086230 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.085958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086230 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.085990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086371 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086371 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086461 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086388    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086461 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086461 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086449    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086560 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086614 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.086759 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.086733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.130159 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130109    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 01:04:25.130159 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130114    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 01:04:25.130334 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130116    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 01:04:25.130334 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130158    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 01:04:25.130334 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130200    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 01:04:25.130334 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130206    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 01:04:25.130334 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130285    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7pd7l\""
Oct 31 01:04:25.130334 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130205    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 01:04:25.130504 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.130417    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 01:04:25.131653 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.131635    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:04:25.187643 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.187810 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.187810 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.187810 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.187810 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.188007 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.188007 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.187942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188233    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188385    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.188632    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.189481 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.189184    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.191119 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.191083    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.191327 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.191306    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.191696 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.191670    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.195537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.193231    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.195537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.193447    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.195537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.193647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.195537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.193657    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.195537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.193797    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.195537 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.193849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.226273 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.226248    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 01:04:25.231077 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.231038    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.246292 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.246266    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.334906 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.334827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:25.407617 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.407588    2574 generic.go:356] "Generic (PLEG): container finished" podID="43cb7dd9-ef71-43ac-8631-84d0ee80468a" containerID="34573608fc6a60d43c7d6c730336ba52009a7223f07cc550fc3497a49720fbf1" exitCode=0
Oct 31 01:04:25.407750 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.407630    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbjh9" event={"ID":"43cb7dd9-ef71-43ac-8631-84d0ee80468a","Type":"ContainerDied","Data":"34573608fc6a60d43c7d6c730336ba52009a7223f07cc550fc3497a49720fbf1"}
Oct 31 01:04:25.820122 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:25.820085    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:04:25.847598 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:25.847568    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d07da0_09ea_4f37_a5e4_8690f2c3882f.slice/crio-7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05 WatchSource:0}: Error finding container 7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05: Status 404 returned error can't find the container with id 7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05
Oct 31 01:04:26.417326 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:26.417286    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05"}
Oct 31 01:04:26.420153 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:26.420093    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbjh9" event={"ID":"43cb7dd9-ef71-43ac-8631-84d0ee80468a","Type":"ContainerStarted","Data":"947ff6fb93f6efa9ecd300360c1c96d68e9f545c518d519dec6ed01002d10370"}
Oct 31 01:04:26.420153 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:26.420154    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fbjh9" event={"ID":"43cb7dd9-ef71-43ac-8631-84d0ee80468a","Type":"ContainerStarted","Data":"462f3131b3e89f5a843603b2276c4a33dcb0254369daf2d9f3985784a42f5326"}
Oct 31 01:04:26.546319 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:26.546258    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fbjh9" podStartSLOduration=3.057224646 podStartE2EDuration="4.546240128s" totalImagesPullingTime="1.402557613s" totalInitContainerRuntime="86.457869ms" isStatefulPod=true podCreationTimestamp="2026-10-31 01:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:26.545019848 +0000 UTC m=+61.002717596" watchObservedRunningTime="2026-10-31 01:04:26.546240128 +0000 UTC m=+61.003937874"
Oct 31 01:04:27.375294 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:27.375273    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:27.427979 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:27.427950    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"}
Oct 31 01:04:28.299845 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.299815    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"]
Oct 31 01:04:28.301907 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.301887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314117 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314092    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314240 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-metrics-client-ca\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314240 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314333 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314367 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-serving-certs-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314407 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314380    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-federate-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314515 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314491    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st64n\" (UniqueName: \"kubernetes.io/projected/b865ce7c-d25a-4154-a2cd-3136b2937143-kube-api-access-st64n\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.314597 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.314522    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-trusted-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.354919 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.354894    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 01:04:28.355192 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.355175    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 01:04:28.355593 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.355578    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 01:04:28.405078 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.405049    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 01:04:28.405078 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.405050    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4fbhm\""
Oct 31 01:04:28.405078 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.405050    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 01:04:28.415489 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415576 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-serving-certs-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415576 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-federate-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415576 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st64n\" (UniqueName: \"kubernetes.io/projected/b865ce7c-d25a-4154-a2cd-3136b2937143-kube-api-access-st64n\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415717 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-trusted-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415717 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415717 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-metrics-client-ca\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.415717 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.415690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.416437 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.416346    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-metrics-client-ca\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.416549 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.416400    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-serving-certs-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.418272 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.418249    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-federate-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.418358 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.418273    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.418358 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.418292    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.418429 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.418401    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.433060 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.433022    2574 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a" exitCode=0
Oct 31 01:04:28.433408 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.433061    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"}
Oct 31 01:04:28.455798 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.455775    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 01:04:28.456644 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.456620    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-trusted-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.457165 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.457128    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"]
Oct 31 01:04:28.473106 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.473087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st64n\" (UniqueName: \"kubernetes.io/projected/b865ce7c-d25a-4154-a2cd-3136b2937143-kube-api-access-st64n\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:28.613107 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:28.613028    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:29.042262 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:29.042226    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"]
Oct 31 01:04:29.047814 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:29.047786    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb865ce7c_d25a_4154_a2cd_3136b2937143.slice/crio-c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d WatchSource:0}: Error finding container c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d: Status 404 returned error can't find the container with id c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d
Oct 31 01:04:29.439041 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:29.438958    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv" event={"ID":"b865ce7c-d25a-4154-a2cd-3136b2937143","Type":"ContainerStarted","Data":"c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d"}
Oct 31 01:04:31.040536 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.040509    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:31.040930 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.040594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:31.106409 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.106381    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:04:31.106504 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.106458    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:04:31.114087 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.114027    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:31.114466 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.114419    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:31.144321 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.141433    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:31.193830 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.193798    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:04:31.253501 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.253464    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:04:31.266212 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.266178    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:31.277191 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.277163    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:04:31.286423 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.286367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:31.291667 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.291643    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:04:31.299948 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.299921    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:31.448887 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.448858    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"}
Oct 31 01:04:31.448993 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.448918    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"}
Oct 31 01:04:31.448993 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.448942    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"}
Oct 31 01:04:31.448993 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.448967    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"}
Oct 31 01:04:31.449092 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.448995    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"}
Oct 31 01:04:31.481056 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.481021    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:04:31.490355 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.490334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:31.669497 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.669447    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vrg94"]
Oct 31 01:04:31.685852 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:31.685815    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549daf04_71a7_40cd_970b_0d14584c26a4.slice/crio-a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0 WatchSource:0}: Error finding container a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0: Status 404 returned error can't find the container with id a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0
Oct 31 01:04:31.717456 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.717420    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tmx2h"]
Oct 31 01:04:31.720678 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:31.720647    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef840aa5_db4c_4422_a2d4_702a5c8bdcb0.slice/crio-0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180 WatchSource:0}: Error finding container 0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180: Status 404 returned error can't find the container with id 0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180
Oct 31 01:04:31.885328 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:31.885240    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7znmf"]
Oct 31 01:04:31.889438 ip-10-0-140-164 kubenswrapper[2574]: W1031 01:04:31.889402    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa45a8e_78a1_4cce_96e8_5ba324073c61.slice/crio-e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f WatchSource:0}: Error finding container e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f: Status 404 returned error can't find the container with id e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f
Oct 31 01:04:32.454149 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.454095    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7znmf" event={"ID":"2fa45a8e-78a1-4cce-96e8-5ba324073c61","Type":"ContainerStarted","Data":"e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f"}
Oct 31 01:04:32.455623 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.455577    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tmx2h" event={"ID":"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0","Type":"ContainerStarted","Data":"0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180"}
Oct 31 01:04:32.459718 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.459679    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerStarted","Data":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"}
Oct 31 01:04:32.462207 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.462178    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv" event={"ID":"b865ce7c-d25a-4154-a2cd-3136b2937143","Type":"ContainerStarted","Data":"0fa110de7db84d16f745d7c0c9ad7aea77233a7ab09bf6d2ab4ff4e630f54019"}
Oct 31 01:04:32.462350 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.462231    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv" event={"ID":"b865ce7c-d25a-4154-a2cd-3136b2937143","Type":"ContainerStarted","Data":"7d5b84649fff84f32b875d25a432a2c5cf029c525077772d3779393b0b74c707"}
Oct 31 01:04:32.462350 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.462258    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv" event={"ID":"b865ce7c-d25a-4154-a2cd-3136b2937143","Type":"ContainerStarted","Data":"f1dc02ffdfd88845c596c8ae38faf9646e77832875208ff5f59bd9dec0e345a1"}
Oct 31 01:04:32.463427 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.463401    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrg94" event={"ID":"549daf04-71a7-40cd-970b-0d14584c26a4","Type":"ContainerStarted","Data":"a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0"}
Oct 31 01:04:32.591814 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.591722    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.51038298 podStartE2EDuration="8.591703219s" totalImagesPullingTime="5.069481242s" totalInitContainerRuntime="11.838997ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:32.590490812 +0000 UTC m=+67.048188581" watchObservedRunningTime="2026-10-31 01:04:32.591703219 +0000 UTC m=+67.049400965"
Oct 31 01:04:32.717874 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:32.717807    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv" podStartSLOduration=2.331479241 podStartE2EDuration="4.717788269s" totalImagesPullingTime="2.386309028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:32.716304674 +0000 UTC m=+67.174002424" watchObservedRunningTime="2026-10-31 01:04:32.717788269 +0000 UTC m=+67.175486016"
Oct 31 01:04:34.476524 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:34.476484    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrg94" event={"ID":"549daf04-71a7-40cd-970b-0d14584c26a4","Type":"ContainerStarted","Data":"ca71e3e567b24b1b3fa85577046e6684e1716105dd2886031fd263b7678127f2"}
Oct 31 01:04:34.476990 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:34.476555    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vrg94" event={"ID":"549daf04-71a7-40cd-970b-0d14584c26a4","Type":"ContainerStarted","Data":"41e88806d6000b0679f1ec1ba2b7172fe581fc0c329d4880714bb1931b106ed4"}
Oct 31 01:04:34.596896 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:34.595452    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vrg94" podStartSLOduration=66.791319524 podStartE2EDuration="1m8.595432608s" totalImagesPullingTime="1.804113084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:34.593393677 +0000 UTC m=+69.051091444" watchObservedRunningTime="2026-10-31 01:04:34.595432608 +0000 UTC m=+69.053130356"
Oct 31 01:04:37.496092 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:37.496052    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7znmf" event={"ID":"2fa45a8e-78a1-4cce-96e8-5ba324073c61","Type":"ContainerStarted","Data":"cd0f0b417e440581dce013bfefd3362190aa0f460aa1697b395a6cef45b17dcf"}
Oct 31 01:04:37.496544 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:37.496199    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:38.507178 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:38.505867    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tmx2h" event={"ID":"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0","Type":"ContainerStarted","Data":"8bf92a0e414d1a0c1c117ed3f802666fc94a0f85eef23b1964e96912c95da1d5"}
Oct 31 01:04:38.617184 ip-10-0-140-164 kubenswrapper[2574]: I1031 01:04:38.617052    2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:04:38.617281 ip-10-0-140-164 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 01:04:38.627821 ip-10-0-140-164 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 01:04:38.628014 ip-10-0-140-164 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 01:04:38.628067 ip-10-0-140-164 systemd[1]: kubelet.service: Consumed 3.988s CPU time.
Oct 31 01:04:38.636972 ip-10-0-140-164 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 01:04:38.735437 ip-10-0-140-164 kubenswrapper[5419]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:04:38.735437 ip-10-0-140-164 kubenswrapper[5419]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 01:04:38.735437 ip-10-0-140-164 kubenswrapper[5419]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:04:38.735437 ip-10-0-140-164 kubenswrapper[5419]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 01:04:38.737916 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737808    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.737916 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737921    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737926    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737931    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737934    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737936    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737939    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737942    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737944    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737947    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737950    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737952    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737955    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737958    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737960    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737963    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737966    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737968    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737971    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737973    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737983    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.738037 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737986    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737989    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737991    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.737997    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738004    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738010    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738015    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738020    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738025    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738031    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738035    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738040    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738044    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738049    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738053    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738057    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738061    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738065    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738069    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.738644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738072    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738075    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738078    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738080    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738083    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738087    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738091    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738095    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738097    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738100    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738102    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738105    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738107    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738110    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738113    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738116    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738119    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738121    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738124    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.739265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738126    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738129    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738132    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738153    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738158    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738162    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738165    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738168    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738171    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738174    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738177    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738179    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738182    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738186    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738189    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738192    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738194    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738197    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738199    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738202    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.739875 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738204    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738207    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738209    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738212    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738214    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738217    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738220    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738223    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738225    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738228    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738231    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738233    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738236    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738238    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738241    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738244    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738249    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738253    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738257    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.740472 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738261    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738266    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738269    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738271    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738275    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738278    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738281    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.738284    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739220    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739228    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739231    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739234    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739237    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739240    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739242    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739245    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739247    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739250    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739252    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739255    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.741048 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739258    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739261    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739263    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739266    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739269    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739271    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739274    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739277    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739281    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739285    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739289    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739293    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739297    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739302    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739306    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739311    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739315    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739320    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739326    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.741690 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739333    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739344    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739348    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739351    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739353    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739356    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739360    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739364    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739367    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739370    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739372    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739375    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739380    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739383    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739385    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739388    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739390    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739393    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739395    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.742372 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739398    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739402    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739405    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739408    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739410    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739413    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739415    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739418    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739420    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739423    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739426    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739429    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739431    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739434    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739436    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739440    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739442    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739445    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739447    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739451    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.742929 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739453    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739456    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739458    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739461    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739463    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739467    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739470    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739472    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739475    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739477    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739480    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739483    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739485    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739489    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739492    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739494    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739497    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739499    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739502    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739505    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.743647 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739507    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739510    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739512    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739515    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739519    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739521    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739524    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739527    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739530    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739533    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739535    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739538    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739540    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739543    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739545    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.739547    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740202    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740213    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740216    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.744421 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740219    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740221    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740224    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740227    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740230    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740232    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740235    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740238    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740241    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740243    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740248    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740251    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740254    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740257    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740259    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740262    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740264    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740267    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740269    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.745017 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740272    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740274    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740278    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740281    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740283    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740286    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740289    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740291    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740294    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740296    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740299    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740303    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740305    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740308    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740310    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740313    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740315    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740318    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740321    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740324    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.745605 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740327    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740329    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740332    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740334    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740337    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740340    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740342    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740345    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740348    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740350    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740353    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740356    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740359    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740361    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740363    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740367    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740371    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740375    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740379    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.746215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740383    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740387    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740391    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740395    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740401    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740405    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740409    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740413    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740417    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740421    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740424    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740427    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740431    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740433    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740436    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740440    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740445    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740448    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740451    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740453    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.746848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740456    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740459    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740461    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740464    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740467    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740470    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740472    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740475    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740478    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740481    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740484    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740487    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740490    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740492    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740495    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740497    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740501    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740503    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740506    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.747587 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740508    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740511    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740513    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740516    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740520    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.740523    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740642    5419 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740653    5419 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740662    5419 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740667    5419 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740671    5419 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740675    5419 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740679    5419 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740684    5419 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740688    5419 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740691    5419 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740694    5419 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740697    5419 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740700    5419 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740703    5419 flags.go:64] FLAG: --cgroup-root=""
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740706    5419 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740709    5419 flags.go:64] FLAG: --client-ca-file=""
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740712    5419 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 01:04:38.748186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740715    5419 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740720    5419 flags.go:64] FLAG: --cluster-domain=""
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740722    5419 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740725    5419 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740728    5419 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740732    5419 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740739    5419 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740742    5419 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740745    5419 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740749    5419 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740756    5419 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740759    5419 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740762    5419 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740767    5419 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740772    5419 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740779    5419 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740784    5419 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740788    5419 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740793    5419 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740798    5419 flags.go:64] FLAG: --enable-server="true"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740803    5419 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740814    5419 flags.go:64] FLAG: --event-burst="100"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740820    5419 flags.go:64] FLAG: --event-qps="50"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740825    5419 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 01:04:38.748907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740830    5419 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740833    5419 flags.go:64] FLAG: --eviction-hard=""
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740838    5419 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740841    5419 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740844    5419 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740847    5419 flags.go:64] FLAG: --eviction-soft=""
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740850    5419 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740853    5419 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740856    5419 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740860    5419 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740863    5419 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740866    5419 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740869    5419 flags.go:64] FLAG: --feature-gates=""
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740874    5419 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740877    5419 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740880    5419 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740884    5419 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740887    5419 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740892    5419 flags.go:64] FLAG: --help="false"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740895    5419 flags.go:64] FLAG: --hostname-override="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740898    5419 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740901    5419 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740904    5419 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 01:04:38.749756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740907    5419 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740910    5419 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740913    5419 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740916    5419 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740919    5419 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740921    5419 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740924    5419 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740927    5419 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740930    5419 flags.go:64] FLAG: --kube-reserved=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740933    5419 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740936    5419 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740939    5419 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740941    5419 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740944    5419 flags.go:64] FLAG: --lock-file=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740947    5419 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740950    5419 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740953    5419 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740960    5419 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740964    5419 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740970    5419 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740975    5419 flags.go:64] FLAG: --logging-format="text"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740980    5419 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740984    5419 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740987    5419 flags.go:64] FLAG: --manifest-url=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740990    5419 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 01:04:38.750775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740994    5419 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.740997    5419 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741002    5419 flags.go:64] FLAG: --max-pods="110"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741006    5419 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741010    5419 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741014    5419 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741018    5419 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741023    5419 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741026    5419 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741028    5419 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741036    5419 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741039    5419 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741042    5419 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741046    5419 flags.go:64] FLAG: --pod-cidr=""
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741049    5419 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741052    5419 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741055    5419 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741058    5419 flags.go:64] FLAG: --port="10250"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741061    5419 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741064    5419 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0afe2e76924e24979"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741068    5419 flags.go:64] FLAG: --qos-reserved=""
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741071    5419 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741074    5419 flags.go:64] FLAG: --register-node="true"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741077    5419 flags.go:64] FLAG: --register-with-taints=""
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741081    5419 flags.go:64] FLAG: --registry-burst="10"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741083    5419 flags.go:64] FLAG: --registry-qps="5"
Oct 31 01:04:38.751495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741086    5419 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741090    5419 flags.go:64] FLAG: --reserved-memory=""
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741094    5419 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741097    5419 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741100    5419 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741102    5419 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741106    5419 flags.go:64] FLAG: --runonce="false"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741109    5419 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741113    5419 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741116    5419 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741119    5419 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741122    5419 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741125    5419 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741128    5419 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741131    5419 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741155    5419 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741160    5419 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741164    5419 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741169    5419 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741177    5419 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741181    5419 flags.go:64] FLAG: --system-cgroups=""
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741184    5419 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741192    5419 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741197    5419 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741201    5419 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 01:04:38.752335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741208    5419 flags.go:64] FLAG: --tls-min-version=""
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741212    5419 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741217    5419 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741220    5419 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741223    5419 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741226    5419 flags.go:64] FLAG: --v="2"
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741231    5419 flags.go:64] FLAG: --version="false"
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741235    5419 flags.go:64] FLAG: --vmodule=""
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741239    5419 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741243    5419 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741443    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741452    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741457    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741461    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741465    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741475    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741481    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741485    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741488    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741491    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741494    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.753030 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741497    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741499    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741502    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741505    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741507    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741511    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741514    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741517    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741519    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741522    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741525    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741527    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741530    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741533    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741535    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741538    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741541    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741543    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741546    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741549    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.753640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741551    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741555    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741558    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741560    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741563    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741565    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741569    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741572    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741574    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741577    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741579    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741582    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741585    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741587    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741589    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741592    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741594    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741598    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741600    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.754247 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741603    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741606    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741608    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741612    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741615    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741617    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741620    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741622    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741625    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741627    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741631    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741634    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741636    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741639    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741641    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741646    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741651    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741655    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741659    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741664    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.754956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741669    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741673    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741677    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741682    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741685    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741687    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741690    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741693    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741696    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741699    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741702    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741705    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741708    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741711    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741714    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741716    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741719    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741721    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741745    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741748    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.755675 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741751    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741753    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741757    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741761    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741764    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741767    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741770    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741772    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741775    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741778    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741781    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741786    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741789    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741793    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741798    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.756402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741802    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.741849    5419 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741901    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741907    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741912    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741918    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741922    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741926    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741932    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741937    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741942    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741947    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741952    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741956    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741960    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741963    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.756912 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741966    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741969    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741971    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741974    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741977    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741980    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741982    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741985    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741987    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741990    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741993    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.741996    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742000    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742003    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742006    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742008    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742011    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742013    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742016    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742021    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.757491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742025    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742029    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742034    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742039    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742045    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742050    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742053    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742056    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742059    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742064    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742068    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742072    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742077    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742081    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742085    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742090    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742094    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742097    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742100    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.758130 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742102    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742105    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742107    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742110    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742113    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742120    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742124    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742126    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742129    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742132    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742152    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742156    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742160    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742165    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742168    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742171    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742175    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742182    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742186    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.758859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742191    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742197    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742201    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742205    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742210    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742214    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742219    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742223    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742227    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742232    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742236    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742240    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742245    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742248    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742251    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742253    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742256    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742259    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742263    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742266    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.759539 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742269    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742271    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742274    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742277    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742279    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742282    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742285    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742288    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742290    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742293    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742298    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742300    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742303    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.742306    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.747096    5419 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.747111    5419 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.747133    5419 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.747156    5419 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747677    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.760248 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747694    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747700    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747705    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747716    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747720    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747725    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747729    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747733    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747738    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747742    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747746    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747751    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747755    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747760    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747764    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747768    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747777    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747782    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747786    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.760878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747790    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747797    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747804    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747809    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747814    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747819    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747823    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747828    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747832    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747874    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747879    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747884    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747888    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747893    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747897    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747901    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747905    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747910    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747914    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.761470 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747919    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747923    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747927    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747936    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747941    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747945    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747949    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747954    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747958    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747963    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747967    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747972    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747976    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747980    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747984    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747993    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.747997    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748001    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748005    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748009    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.762005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748014    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748018    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748022    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748053    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748057    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748062    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748066    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748070    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748079    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748083    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748087    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748091    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748095    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748125    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748130    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748160    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748165    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748169    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748173    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.762694 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748177    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748181    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748190    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748195    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748199    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748204    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748208    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748213    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748217    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748221    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748224    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748229    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748233    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748238    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748246    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748251    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748255    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748289    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748296    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.763540 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748302    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748306    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748310    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748315    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748320    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748323    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748328    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748332    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.748341    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.748400    5419 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749168    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749182    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749188    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749192    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749196    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749200    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.764494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749205    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749209    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749213    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749217    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749224    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749231    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749236    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749242    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749248    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749252    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749258    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749263    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749267    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749272    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749276    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749281    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749286    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749291    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749295    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749299    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.765042 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749303    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749307    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749311    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749315    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749320    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749325    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749329    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749333    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749337    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749341    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749345    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749350    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749355    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749359    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749364    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749368    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749372    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749377    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749401    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749407    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.765658 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749411    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749415    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749419    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749424    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749428    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749432    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749438    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749442    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749447    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749451    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749455    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749470    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749474    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749478    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749482    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749486    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749490    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749495    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749500    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.766418 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749504    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749508    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749512    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749517    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749521    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749596    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749601    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749606    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749611    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749615    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749620    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749624    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749628    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749632    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749636    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749640    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749644    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749649    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749653    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.766985 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749658    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749662    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749666    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749670    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749674    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749678    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749682    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749686    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749691    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749694    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749698    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749702    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749707    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749711    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749715    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749719    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749723    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749727    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749733    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.767660 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749739    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749746    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749751    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.749797    5419 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749852    5419 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749860    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749865    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749869    5419 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749873    5419 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749879    5419 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749885    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749890    5419 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749894    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749898    5419 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749903    5419 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749907    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 01:04:38.768307 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749912    5419 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749916    5419 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749920    5419 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749924    5419 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749928    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749932    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749936    5419 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749943    5419 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749949    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749953    5419 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749957    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749961    5419 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749965    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749969    5419 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749973    5419 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749977    5419 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749981    5419 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749985    5419 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749989    5419 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749993    5419 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 01:04:38.768840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.749997    5419 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750001    5419 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750006    5419 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750010    5419 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750014    5419 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750018    5419 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750022    5419 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750026    5419 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750031    5419 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750035    5419 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750039    5419 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750043    5419 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750048    5419 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750053    5419 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750057    5419 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750061    5419 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750065    5419 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750069    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750073    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750077    5419 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 01:04:38.769589 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750081    5419 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750085    5419 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750089    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750093    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750097    5419 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750101    5419 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750105    5419 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750109    5419 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750113    5419 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750117    5419 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750121    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750127    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750131    5419 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750151    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750155    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750160    5419 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750165    5419 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750169    5419 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750173    5419 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 01:04:38.770309 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750177    5419 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750181    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750185    5419 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750189    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750193    5419 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750197    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750201    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750205    5419 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750209    5419 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750213    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750217    5419 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750221    5419 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750225    5419 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750229    5419 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750233    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750237    5419 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750241    5419 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750244    5419 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750248    5419 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 01:04:38.771040 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750252    5419 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750256    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750260    5419 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750264    5419 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750268    5419 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750272    5419 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750276    5419 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750280    5419 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750285    5419 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750289    5419 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750293    5419 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750298    5419 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750302    5419 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750306    5419 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750310    5419 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:38.750321    5419 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.750399    5419 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.751965    5419 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.752033    5419 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 01:04:38.771863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.752320    5419 server.go:1014] "Starting client certificate rotation"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.752458    5419 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.752511    5419 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:58:26 +0000 UTC" deadline="2028-07-05 10:14:59.57604859 +0000 UTC"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.752543    5419 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14721h10m20.823507373s"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.753122    5419 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.754920    5419 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.756358    5419 log.go:25] "Validated CRI v1 runtime API"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.758799    5419 log.go:25] "Validated CRI v1 image API"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.759962    5419 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 01:04:38.772715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.762254    5419 fs.go:135] Filesystem UUIDs: map[30bf5f50-d927-4b49-8fe4-926af04c3f16:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f50b57d3-a57c-48d5-b6ce-3c76c2fc40cb:/dev/nvme0n1p4]
Oct 31 01:04:38.773101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.762279    5419 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/111a48d2-3e5c-4c68-8215-98b0cfc5b301/volumes/kubernetes.io~projected/kube-api-access-tngvl:{mountpoint:/var/lib/kubelet/pods/111a48d2-3e5c-4c68-8215-98b0cfc5b301/volumes/kubernetes.io~projected/kube-api-access-tngvl major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/120efa71-c477-4936-b599-e9f81f035a25/volumes/kubernetes.io~projected/kube-api-access-b5v92:{mountpoint:/var/lib/kubelet/pods/120efa71-c477-4936-b599-e9f81f035a25/volumes/kubernetes.io~projected/kube-api-access-b5v92 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/198bdfe6-2b3f-4426-a0cc-6568d22b139d/volumes/kubernetes.io~projected/kube-api-access-qtv7v:{mountpoint:/var/lib/kubelet/pods/198bdfe6-2b3f-4426-a0cc-6568d22b139d/volumes/kubernetes.io~projected/kube-api-access-qtv7v major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/198bdfe6-2b3f-4426-a0cc-6568d22b139d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/198bdfe6-2b3f-4426-a0cc-6568d22b139d/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c9dd849-9bfc-44bf-8eba-b8a08677c824/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1c9dd849-9bfc-44bf-8eba-b8a08677c824/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~projected/kube-api-access-w99sz:{mountpoint:/var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~projected/kube-api-access-w99sz major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fa45a8e-78a1-4cce-96e8-5ba324073c61/volumes/kubernetes.io~projected/kube-api-access-nbt4j:{mountpoint:/var/lib/kubelet/pods/2fa45a8e-78a1-4cce-96e8-5ba324073c61/volumes/kubernetes.io~projected/kube-api-access-nbt4j major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~projected/kube-api-access-65jc4:{mountpoint:/var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~projected/kube-api-access-65jc4 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/549daf04-71a7-40cd-970b-0d14584c26a4/volumes/kubernetes.io~projected/kube-api-access-x94xd:{mountpoint:/var/lib/kubelet/pods/549daf04-71a7-40cd-970b-0d14584c26a4/volumes/kubernetes.io~projected/kube-api-access-x94xd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/549daf04-71a7-40cd-970b-0d14584c26a4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/549daf04-71a7-40cd-970b-0d14584c26a4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ee4c566-ef1d-42ec-8e17-3de154e4ec69/volumes/kubernetes.io~projected/kube-api-access-mkj5v:{mountpoint:/var/lib/kubelet/pods/6ee4c566-ef1d-42ec-8e17-3de154e4ec69/volumes/kubernetes.io~projected/kube-api-access-mkj5v major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~projected/kube-api-access-9nzj7:{mountpoint:/var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~projected/kube-api-access-9nzj7 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7d6f01-4cbb-4865-931e-86cf217018fe/volumes/kubernetes.io~projected/kube-api-access-jv56d:{mountpoint:/var/lib/kubelet/pods/8e7d6f01-4cbb-4865-931e-86cf217018fe/volumes/kubernetes.io~projected/kube-api-access-jv56d major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96abcc08-0354-41d9-baec-af53dd804fdd/volumes/kubernetes.io~projected/kube-api-access-t8d4p:{mountpoint:/var/lib/kubelet/pods/96abcc08-0354-41d9-baec-af53dd804fdd/volumes/kubernetes.io~projected/kube-api-access-t8d4p major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96abcc08-0354-41d9-baec-af53dd804fdd/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/96abcc08-0354-41d9-baec-af53dd804fdd/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volumes/kubernetes.io~projected/kube-api-access-96mtd:{mountpoint:/var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volumes/kubernetes.io~projected/kube-api-access-96mtd major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa5bbf79-871a-4f48-beea-cb3bf34fbe64/volumes/kubernetes.io~projected/kube-api-access-9w98f:{mountpoint:/var/lib/kubelet/pods/aa5bbf79-871a-4f48-beea-cb3bf34fbe64/volumes/kubernetes.io~projected/kube-api-access-9w98f major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af0e6596-aec8-43f1-9aac-2497c9563bef/volumes/kubernetes.io~projected/kube-api-access-78cfp:{mountpoint:/var/lib/kubelet/pods/af0e6596-aec8-43f1-9aac-2497c9563bef/volumes/kubernetes.io~projected/kube-api-access-78cfp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af0e6596-aec8-43f1-9aac-2497c9563bef/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/af0e6596-aec8-43f1-9aac-2497c9563bef/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~projected/kube-api-access-st64n:{mountpoint:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~projected/kube-api-access-st64n major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda/volumes/kubernetes.io~projected/kube-api-access-g4vd4:{mountpoint:/var/lib/kubelet/pods/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda/volumes/kubernetes.io~projected/kube-api-access-g4vd4 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/alertmanager/9 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/config-reloader/6 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~projected/kube-api-access-pjf75:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~projected/kube-api-access-pjf75 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~projected/tls-assets major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/config-volume major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/web-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de59f004-1680-492e-88f9-3278ed575e50/volumes/kubernetes.io~projected/kube-api-access-lkfbg:{mountpoint:/var/lib/kubelet/pods/de59f004-1680-492e-88f9-3278ed575e50/volumes/kubernetes.io~projected/kube-api-access-lkfbg major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:340 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2248fc36450b9ed7db4eab56cba022e0a8cf1fdddcc9687f247820bca474edea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/da7072a64c4b4382da6af04290527da180fc916ade972c6d4a1432cd175fa646/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f6433700c599672c5bdb7b74e759e9059ab52967a363992577f132f291fb589f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/348302c9e50f67c65c1499a848c2fa37fe9ea9ba2e97217266a21a1169b0dc3a/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/151d35d6e0d7236f4c381457a1b8ec57478b75eb2db4734324a8576ed1cff350/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4613478528eb097c87bc5ce8932c9ade8b114cab3ee6dd896bde4d85bc96ec70/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1bd7d77bfda4617df9601e16007dcfc72eba2654efbdf7303b7f3336167d49d7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4ee1fd950fd9caf1af82884ed174c040e79abdb7ade77daf5c684c397cb6bf98/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/233f9b6d44acba6b7a255e817e4a5363441790c511560670694a22442e57f2cc/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/5e5081d178cb142b5ad16bdc8d2de15cd5aac7926441b03e5bf76acc990c93f7/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e19ac7fbcb3ee133fe95e1aee456c8f5b0def164f5fce258d0793b13b102e40a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/27febbb036d4c62466e19c56d9d88aaf7f302d86115b9f96e52767acc88df4fe/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ed0d2fffae613ff40e018cdd022f306474e87ef5f22e35126b1466aebcc7a936/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a9e3b4fadb172c64ba12bef023cf5db3507cda7fb0e82424f95ea146833bc02c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/91a107d405373f9772cd673e266c6bc2d73bd41152d6c2b3c0b5214b418f113d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/6c21b2e8f27ec2384f62ed898c4f7e99a15cd7b06082ff36c6c7ccd974d3f753/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1de9afe87519c7b2cf833efd1d1214466df93277f887d9e5b56559b3572a3d3e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f647f75d35029faa13850aaebd361a9ae6391a7d1c28293ef75df435f78a55c0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/533a02692bde22c11f3fb32dc803e474d190636d9f9a19c19a54e58f331ea0bc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/32b1ec7621a2c6ede6b62aec000d0a7b1b0e4be3fd7d8ba5f67148e22f4c5330/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f574e76e860781098e56b17894b525e989ac3c7704e0e5b24449db163b40fba2/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3c3c9dd4718db7a5d71b12834186707f3fd1cb7985d1aeffbaab1d5e642702fd/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f313cecc25d9e163f8be9d09687973d70181a3521536eaa4619f5da4b930309d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/8b8702e016873112dd1d839623dbe9e58ea37c2e0d6bf6b8fdd800524f58fb37/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/c6ab33f3702badf0db7a17f88522c4aac4f9f96d35886b53091f8bb81a4c93cd/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/a2c31bd0b2d304ac6990bad2ad0a74026ba48f19d2316d6176207e3a5d68f47e/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3fb23956d5da8fc031d176eeb9de2dcb08b2574ddda46e5bbdb9bcfbb6231dc9/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/75b42997c0b928544b9069c423e0dbf6f97b04a0f8b768e5b61b6fa71954c855/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/0e3829fc8b4977873d15d092feae136c06d9c0a2736a289d9bcda01a13cd9d31/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/dec8e960d52c6b371fb151a0d2a8b4b26ec8bf87c4a18ccd1ca52a025a96009c/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/a8cc96da9032c3c4434abca849b82fddde25f1eae73fe2f4a32a34d805dc9b44/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/10fe950f2feb326cc03fd25995765fff221641345994a872e6b9cb22a8d9b538/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/eae55efb5cba2d5897b7b5bddce927365c20a619423298770c93c3b57cb205cf/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/7261d8479a08e75d1479b5b9815043e98ce12e4d6f57350c98ecde5573cc34fa/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/3b104fd8af43ff16223dcd653e65246f9518f0ef3fd9be6688f887e04f44c009/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/5a2e90abf5ea882b66080c12172d998f4cb9f0af2dd6389d2b83d367aae1583e/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/8df3d2d2ef2366e2311a378aad75f9553e6d08571ba4ff591288a7b6e040a471/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/f8fe9fd9af21ff21e4e9e48fc91289a87e128f20f76c923eacb4872cb9e8ad44/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/c6598d0dbd747595c4d73940ae59a2fa7f10f8e516befbfcc594207257710bd5/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/fce757e6a82dda7579f05d0f5457b52fb7b39a28c4ad64326616c26d494446d1/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/dcf9b7a7f78aa14fa5b70960de24a4d97c236bfb6d84b49e6105d81837695f2f/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/d4dfc96184b47fb847a06e18180a952752418acf574d456757ce74b07d53919f/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/884c1421a3ae75c91f57423d28c9b2c622c5e7beb60b2d5d743eeea8fad692e1/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/0b93d6217953c39dd307d499b04f81b8812b4b5725debabf74adf4ba83eab486/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/a7906f25b56e800e8bad91be0a945cacbf97d593db9831c87e01cc7416f4ec81/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/8a9bf84eda23445c412c45fb80ac9e256a5594a5613a6ba6958ce3343457bb4f/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/f2bc96f702176ba3fb101b6ccbbee77aa3f8d90d271e1a85d761c2ebb6414896/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/04311766bdf324f9e16a3d03cf97957baeb648646160860840afebfe5a33816b/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/a963ef71e35716a888dd13a7e20bffc9a03219b79c0b9adb5572ecc22d16fc95/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/02e9188eb246deebf9f53c8497eff8d6c3fa7628659055095513c3ddbe4abb22/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/649094259a2acc905a7768c0da8588acb564031c31fa4808aa745a1ad4c02a76/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/9e336d2f5fdfa6ae43c89d5d45a839b99bdd77957280748de298ed53b5679405/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/8107cdf859add270019c543d78442b8740daded345289df36f09058511600c73/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/3b613d5dbb64ebc874a085e643c3a059852f0fe4633b28e2094c3c706f500a6f/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/b1335a4fba8399ddf64f675caa5ecc5aae13c0c1ef6f0fe0ad43fd111a59b9e1/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/c7f2082ce387216782bcb1904e4fe44c07288d13a20f9505539d1ded56d5fcfe/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/6eaa0b602ebdd8a880b176629eb6c0a1f425938c3f20a22724fd3909bbec2823/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9b9b2a5caf29a3057c5ecd76762fc4fe88ee68a124d389fba4e2f8c82da07686/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/abd6c3ed2b1fc2a01fe5a8b6d95f9f8bf9b9459762b9317820562d7d5dd698b2/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e9b9fa0491eba951418fc4e73342ddb14ffcbd48440c5cd89a760100a204784b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b2a9bfc72c22b8119a9f09f4ed39e9b8e4deb9017a7fb147e35e04f675d7c5a4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a9723a05cd80dae6b69babd9ac17ceba8e62ff64042949e994f7cac7343c06c1/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d5dff49d90b18ba97761b02f1c0cdf97858055ca72493722b8e60eb829ed69a1/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/567d752cff110c110796ba4f7e59894075fce09a1bd33b87156279ecc6f3f448/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f3aafd09d8a7e76f6536cc1789570faa2f204efb7924af5595c727f384d70c56/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 01:04:38.773420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.771989    5419 manager.go:217] Machine: {Timestamp:2026-10-31 01:04:38.77139715 +0000 UTC m=+0.072353720 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099744 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e8dbf09d73ffedaff07d053a730ce SystemUUID:ec2e8dbf-09d7-3ffe-daff-07d053a730ce BootID:699e8e10-9adc-4ca2-ae32-cd9c7f960a48 Filesystems:[{Device:/var/lib/kubelet/pods/549daf04-71a7-40cd-970b-0d14584c26a4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:336 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa5bbf79-871a-4f48-beea-cb3bf34fbe64/volumes/kubernetes.io~projected/kube-api-access-9w98f DeviceMajor:0 DeviceMinor:53 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~projected/kube-api-access-st64n DeviceMajor:0 DeviceMinor:308 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:274 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:340 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:273 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~projected/kube-api-access-pjf75 DeviceMajor:0 DeviceMinor:289 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e55edbaded337c36e3e8387a735b962431fcaf1c3fb29abecfc5791308c9a14f/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/de59f004-1680-492e-88f9-3278ed575e50/volumes/kubernetes.io~projected/kube-api-access-lkfbg DeviceMajor:0 DeviceMinor:52 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e7f2744f80c3c364b7e612bd3bfe84caea291ae1244957d313f9a43096e7aed/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a872dd06574682bd870a93034616e93b13527b444247125ba68819ea990ab9f0/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volumes/kubernetes.io~projected/kube-api-access-96mtd DeviceMajor:0 DeviceMinor:58 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec97cb10b7efc6477b4b806f8b85f360cd35fb55be860b9adbe4dad40ad1fd7b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bae219737fb05ecb9c777948ff67a5436ac3fcddb71d2c36a223a56e8495d0a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/af0e6596-aec8-43f1-9aac-2497c9563bef/volumes/kubernetes.io~projected/kube-api-access-78cfp DeviceMajor:0 DeviceMinor:203 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f43edce57c7bd7f676a3abfd842e63b804f435b90824e0e9b2382a61e3d58607/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda/volumes/kubernetes.io~projected/kube-api-access-g4vd4 DeviceMajor:0 DeviceMinor:56 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7043088295e5dd29a06af989505d1e3dbba592ec34535d46ae3c73f4b114d462/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/198bdfe6-2b3f-4426-a0cc-6568d22b139d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:306 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce0755dd3cc27442b38869b38462aa8899512ead5799a73ac6bbb311a255e2b0/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:282 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:217 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea1d2cbd0741443f1e8f7f340ac987001bc99c767930a0cb906e5fccb2ede458/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:280 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/2fa45a8e-78a1-4cce-96e8-5ba324073c61/volumes/kubernetes.io~projected/kube-api-access-nbt4j DeviceMajor:0 DeviceMinor:341 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ee4c566-ef1d-42ec-8e17-3de154e4ec69/volumes/kubernetes.io~projected/kube-api-access-mkj5v DeviceMajor:0 DeviceMinor:51 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43cb7dd9-ef71-43ac-8631-84d0ee80468a/volumes/kubernetes.io~projected/kube-api-access-65jc4 DeviceMajor:0 DeviceMinor:275 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/1c9dd849-9bfc-44bf-8eba-b8a08677c824/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/120efa71-c477-4936-b599-e9f81f035a25/volumes/kubernetes.io~projected/kube-api-access-b5v92 DeviceMajor:0 DeviceMinor:57 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:287 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17f8a79a68371d9822ecd740c006697666cde1958758ac9ab06e1790b5968630/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/198bdfe6-2b3f-4426-a0cc-6568d22b139d/volumes/kubernetes.io~projected/kube-api-access-qtv7v DeviceMajor:0 DeviceMinor:204 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/549daf04-71a7-40cd-970b-0d14584c26a4/volumes/kubernetes.io~projected/kube-api-access-x94xd DeviceMajor:0 DeviceMinor:59 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:284 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7d6f01-4cbb-4865-931e-86cf217018fe/volumes/kubernetes.io~projected/kube-api-access-jv56d DeviceMajor:0 DeviceMinor:55 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1fb03723eb32947bb01630c2df8e5166aedfbcf568aa0354f28d58ee63c6b87/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af0e6596-aec8-43f1-9aac-2497c9563bef/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:224 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0908b2bbd9a4d88aa1370d738c7bea315ce089a7af0aab295569900f6782ab6/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:282 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2102cc5c692fc9b40cc7930bbf1bb0ea60e54b923781914b92f7417fd0acc00/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:304 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9b7b88037d21c97582f079acd38c34240b9da1c3733d3e9bfe6e8254899122d/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e9d59c68f35c33f4bbbd4b8d3997201fd23cada872f99ffae7c9986a7cfb180/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:281 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:282 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/221a9c21-35c6-4628-87ca-0cc77c4b3595/volumes/kubernetes.io~projected/kube-api-access-w99sz DeviceMajor:0 DeviceMinor:54 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1e61b87cbde0147fb22db96c0a8dc35f3765f4adf04ca574e0711cac223e9e2/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:307 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/b865ce7c-d25a-4154-a2cd-3136b2937143/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:305 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/111a48d2-3e5c-4c68-8215-98b0cfc5b301/volumes/kubernetes.io~projected/kube-api-access-tngvl DeviceMajor:0 DeviceMinor:243 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:285 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/96abcc08-0354-41d9-baec-af53dd804fdd/volumes/kubernetes.io~projected/kube-api-access-t8d4p DeviceMajor:0 DeviceMinor:202 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96abcc08-0354-41d9-baec-af53dd804fdd/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcaba0b62b0f99b2ff082f12247c2ead35615e5082546cf8b49b159be3481c35/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e390179c52a0e02e9a4b70a29d4b2e50f690cb64b29042976fcaac71dd40749/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:282 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cae5eb3-8cac-465d-9b07-f596954cf636/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71865382-46e6-4f6a-bb1f-9b5406439186/volumes/kubernetes.io~projected/kube-api-access-9nzj7 DeviceMajor:0 DeviceMinor:225 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e9d59c68f35c33 MacAddress:2e:b7:a9:7d:da:6d Speed:10000 Mtu:8901} {Name:7525d3711c66a06 MacAddress:66:7e:11:b4:78:2d Speed:10000 Mtu:8901} {Name:a872dd06574682b MacAddress:c6:a7:4f:8c:41:1e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:32:be:3d:42:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:a6:3e:69:2b:9b:87 Speed:0 Mtu:8901} {Name:c9b7b88037d21c9 MacAddress:36:eb:f4:ec:13:89 Speed:10000 Mtu:8901} {Name:d1fb03723eb3294 MacAddress:be:13:b4:58:9f:97 Speed:10000 Mtu:8901} {Name:e0908b2bbd9a4d8 MacAddress:ca:7f:e9:ba:a2:97 Speed:10000 Mtu:8901} {Name:e55edbaded337c3 MacAddress:e2:6b:fd:dd:70:bf Speed:10000 Mtu:8901} {Name:ea1d2cbd0741443 MacAddress:da:d9:ff:16:b6:df Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:32:be:3d:42:3f Speed:0 Mtu:9001} {Name:f43edce57c7bd7f MacAddress:2e:36:96:28:29:3c Speed:10000 Mtu:8901} {Name:fcaba0b62b0f99b MacAddress:ba:cc:e3:4d:97:0a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:c2:b1:5d:89:db:8e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8e:91:66:26:ac:b0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 01:04:38.773420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.772653    5419 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 01:04:38.773420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.772710    5419 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 01:04:38.773710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773409    5419 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 01:04:38.773710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773441    5419 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-164.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 01:04:38.773710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773636    5419 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 01:04:38.773710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773649    5419 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 01:04:38.773710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773663    5419 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:04:38.773710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773684    5419 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 01:04:38.773890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.773806    5419 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 01:04:38.774018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774006    5419 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 01:04:38.774182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774158    5419 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 01:04:38.774237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774222    5419 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 01:04:38.774275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774236    5419 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 01:04:38.774275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774250    5419 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 01:04:38.774275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774263    5419 kubelet.go:405] "Adding apiserver pod source"
Oct 31 01:04:38.774275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.774274    5419 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 01:04:38.775106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.775087    5419 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:04:38.775106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.775103    5419 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:04:38.775703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.775657    5419 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 01:04:38.789082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789063    5419 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 01:04:38.789082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789083    5419 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 01:04:38.789244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789237    5419 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 01:04:38.789471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789450    5419 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 01:04:38.789573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789490    5419 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 01:04:38.789642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789609    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 01:04:38.789642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789628    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 01:04:38.789642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789638    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789653    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789664    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789673    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789682    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789691    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789701    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789711    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789730    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789745    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 01:04:38.789785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789781    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 01:04:38.790367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.789790    5419 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 01:04:38.793636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.793620    5419 server.go:1263] "Started kubelet"
Oct 31 01:04:38.793735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.793702    5419 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 01:04:38.793850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.793801    5419 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 01:04:38.793896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.793872    5419 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 01:04:38.794129 ip-10-0-140-164 systemd[1]: Started Kubernetes Kubelet.
Oct 31 01:04:38.794840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.794493    5419 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 01:04:38.795216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.795196    5419 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 01:04:38.801020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801001    5419 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 01:04:38.801113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801018    5419 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 01:04:38.801113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801103    5419 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 01:04:38.801238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801119    5419 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 01:04:38.801238 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.801123    5419 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-164.us-west-1.compute.internal\" not found"
Oct 31 01:04:38.801238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801192    5419 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 01:04:38.801436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801068    5419 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:58:27 +0000 UTC" deadline="2028-07-01 19:44:40.648484378 +0000 UTC"
Oct 31 01:04:38.801490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.801440    5419 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14634h40m1.847049763s"
Oct 31 01:04:38.803090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803060    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-federate-client-tls" seLinuxMountContext=""
Oct 31 01:04:38.803172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803097    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 01:04:38.803172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803113    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111a48d2-3e5c-4c68-8215-98b0cfc5b301" volumeName="kubernetes.io/empty-dir/111a48d2-3e5c-4c68-8215-98b0cfc5b301-tmp" seLinuxMountContext=""
Oct 31 01:04:38.803172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803127    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fa45a8e-78a1-4cce-96e8-5ba324073c61" volumeName="kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j" seLinuxMountContext=""
Oct 31 01:04:38.803172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803155    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:38.803172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803166    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549daf04-71a7-40cd-970b-0d14584c26a4" volumeName="kubernetes.io/projected/549daf04-71a7-40cd-970b-0d14584c26a4-kube-api-access-x94xd" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803173    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803183    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71865382-46e6-4f6a-bb1f-9b5406439186" volumeName="kubernetes.io/projected/71865382-46e6-4f6a-bb1f-9b5406439186-kube-api-access-9nzj7" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803192    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43cb7dd9-ef71-43ac-8631-84d0ee80468a" volumeName="kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803199    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71865382-46e6-4f6a-bb1f-9b5406439186" volumeName="kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803206    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96abcc08-0354-41d9-baec-af53dd804fdd" volumeName="kubernetes.io/projected/96abcc08-0354-41d9-baec-af53dd804fdd-kube-api-access-t8d4p" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803212    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cae5eb3-8cac-465d-9b07-f596954cf636" volumeName="kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803219    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c9dd849-9bfc-44bf-8eba-b8a08677c824" volumeName="kubernetes.io/configmap/1c9dd849-9bfc-44bf-8eba-b8a08677c824-konnectivity-ca" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803228    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803237    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803250    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="549daf04-71a7-40cd-970b-0d14584c26a4" volumeName="kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803262    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5bbf79-871a-4f48-beea-cb3bf34fbe64" volumeName="kubernetes.io/empty-dir/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-tmp-dir" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803269    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803277    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803285    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803292    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de59f004-1680-492e-88f9-3278ed575e50" volumeName="kubernetes.io/projected/de59f004-1680-492e-88f9-3278ed575e50-kube-api-access-lkfbg" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803300    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43cb7dd9-ef71-43ac-8631-84d0ee80468a" volumeName="kubernetes.io/projected/43cb7dd9-ef71-43ac-8631-84d0ee80468a-kube-api-access-65jc4" seLinuxMountContext=""
Oct 31 01:04:38.803316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803311    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cae5eb3-8cac-465d-9b07-f596954cf636" volumeName="kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-env-overrides" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803324    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cae5eb3-8cac-465d-9b07-f596954cf636" volumeName="kubernetes.io/projected/9cae5eb3-8cac-465d-9b07-f596954cf636-kube-api-access-96mtd" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803334    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803343    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda" volumeName="kubernetes.io/configmap/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-iptables-alerter-script" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803351    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda" volumeName="kubernetes.io/projected/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-kube-api-access-g4vd4" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803358    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803369    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c9dd849-9bfc-44bf-8eba-b8a08677c824" volumeName="kubernetes.io/secret/1c9dd849-9bfc-44bf-8eba-b8a08677c824-agent-certs" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803380    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="221a9c21-35c6-4628-87ca-0cc77c4b3595" volumeName="kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-tmp" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803391    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0e6596-aec8-43f1-9aac-2497c9563bef" volumeName="kubernetes.io/configmap/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803399    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0e6596-aec8-43f1-9aac-2497c9563bef" volumeName="kubernetes.io/empty-dir/af0e6596-aec8-43f1-9aac-2497c9563bef-data-volume" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803406    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0e6596-aec8-43f1-9aac-2497c9563bef" volumeName="kubernetes.io/secret/af0e6596-aec8-43f1-9aac-2497c9563bef-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803414    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803421    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803429    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198bdfe6-2b3f-4426-a0cc-6568d22b139d" volumeName="kubernetes.io/projected/198bdfe6-2b3f-4426-a0cc-6568d22b139d-kube-api-access-qtv7v" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803438    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" volumeName="kubernetes.io/projected/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-kube-api-access-mkj5v" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803452    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96abcc08-0354-41d9-baec-af53dd804fdd" volumeName="kubernetes.io/empty-dir/96abcc08-0354-41d9-baec-af53dd804fdd-tmp-dir" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803460    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cae5eb3-8cac-465d-9b07-f596954cf636" volumeName="kubernetes.io/secret/9cae5eb3-8cac-465d-9b07-f596954cf636-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803467    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803474    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803481    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803488    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43cb7dd9-ef71-43ac-8631-84d0ee80468a" volumeName="kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803495    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71865382-46e6-4f6a-bb1f-9b5406439186" volumeName="kubernetes.io/configmap/71865382-46e6-4f6a-bb1f-9b5406439186-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803506    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="111a48d2-3e5c-4c68-8215-98b0cfc5b301" volumeName="kubernetes.io/projected/111a48d2-3e5c-4c68-8215-98b0cfc5b301-kube-api-access-tngvl" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803518    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" volumeName="kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803530    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af0e6596-aec8-43f1-9aac-2497c9563bef" volumeName="kubernetes.io/projected/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-api-access-78cfp" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803570    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/projected/b865ce7c-d25a-4154-a2cd-3136b2937143-kube-api-access-st64n" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803586    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b865ce7c-d25a-4154-a2cd-3136b2937143" volumeName="kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-client-tls" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803607    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803616    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43cb7dd9-ef71-43ac-8631-84d0ee80468a" volumeName="kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-metrics-client-ca" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803623    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120efa71-c477-4936-b599-e9f81f035a25" volumeName="kubernetes.io/projected/120efa71-c477-4936-b599-e9f81f035a25-kube-api-access-b5v92" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803630    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43cb7dd9-ef71-43ac-8631-84d0ee80468a" volumeName="kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-tls" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803637    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="221a9c21-35c6-4628-87ca-0cc77c4b3595" volumeName="kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-tuned" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803644    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96abcc08-0354-41d9-baec-af53dd804fdd" volumeName="kubernetes.io/secret/96abcc08-0354-41d9-baec-af53dd804fdd-metrics-tls" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803652    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cae5eb3-8cac-465d-9b07-f596954cf636" volumeName="kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-config" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803665    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803672    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef840aa5-db4c-4422-a2d4-702a5c8bdcb0" volumeName="kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803680    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120efa71-c477-4936-b599-e9f81f035a25" volumeName="kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-cni-binary-copy" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803688    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43cb7dd9-ef71-43ac-8631-84d0ee80468a" volumeName="kubernetes.io/empty-dir/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-textfile" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803696    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" volumeName="kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803704    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa5bbf79-871a-4f48-beea-cb3bf34fbe64" volumeName="kubernetes.io/projected/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-kube-api-access-9w98f" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803711    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" volumeName="kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803719    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de59f004-1680-492e-88f9-3278ed575e50" volumeName="kubernetes.io/configmap/de59f004-1680-492e-88f9-3278ed575e50-serviceca" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803726    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="221a9c21-35c6-4628-87ca-0cc77c4b3595" volumeName="kubernetes.io/projected/221a9c21-35c6-4628-87ca-0cc77c4b3595-kube-api-access-w99sz" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803736    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71865382-46e6-4f6a-bb1f-9b5406439186" volumeName="kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803750    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7d6f01-4cbb-4865-931e-86cf217018fe" volumeName="kubernetes.io/projected/8e7d6f01-4cbb-4865-931e-86cf217018fe-kube-api-access-jv56d" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803759    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" volumeName="kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803766    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="120efa71-c477-4936-b599-e9f81f035a25" volumeName="kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-multus-daemon-config" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803777    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="198bdfe6-2b3f-4426-a0cc-6568d22b139d" volumeName="kubernetes.io/secret/198bdfe6-2b3f-4426-a0cc-6568d22b139d-cert" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803790    5419 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96abcc08-0354-41d9-baec-af53dd804fdd" volumeName="kubernetes.io/configmap/96abcc08-0354-41d9-baec-af53dd804fdd-config-volume" seLinuxMountContext=""
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803802    5419 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 01:04:38.803830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.803810    5419 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 01:04:38.807338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.807319    5419 factory.go:55] Registering systemd factory
Oct 31 01:04:38.807338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.807340    5419 factory.go:223] Registration of the systemd container factory successfully
Oct 31 01:04:38.807904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.807888    5419 factory.go:153] Registering CRI-O factory
Oct 31 01:04:38.807982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.807907    5419 factory.go:223] Registration of the crio container factory successfully
Oct 31 01:04:38.807982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.807961    5419 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 01:04:38.808086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.807985    5419 factory.go:103] Registering Raw factory
Oct 31 01:04:38.808086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.808004    5419 manager.go:1199] Started watching for new ooms in manager
Oct 31 01:04:38.809172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.808336    5419 manager.go:319] Starting recovery of all containers
Oct 31 01:04:38.827565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.827507    5419 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 01:04:38.829231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.829207    5419 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 01:04:38.829231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.829229    5419 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 01:04:38.832929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.829505    5419 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 01:04:38.832929 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.829568    5419 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 01:04:38.851805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.851771    5419 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="b86e389df5bc3e46369bf54a37f6867f022ceee764341582da83bc246ca4be45" exitCode=0
Oct 31 01:04:38.851805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.851802    5419 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="5be993e651766de9b9cffda59ca42f29d7c7235884f4bb8598345e4d1090650a" exitCode=0
Oct 31 01:04:38.851805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.851809    5419 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="241b20c97e7aaff3768c61fa2dce7cf396edbc90b1b9c18f9cce9b4b0afd41de" exitCode=0
Oct 31 01:04:38.851805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.851815    5419 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="13026f4aea9f89bc16fe7327d26c9b247ba427e2f02a9a4acb8238ea620213c4" exitCode=0
Oct 31 01:04:38.852114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.851821    5419 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="91b635864c295304f278f100131c3325e63e9f4dd083cd67efcc73f5ec6be8fa" exitCode=0
Oct 31 01:04:38.852114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.851829    5419 generic.go:356] "Generic (PLEG): container finished" podID="6ee4c566-ef1d-42ec-8e17-3de154e4ec69" containerID="cf36fff776abb3dac0b3c7cfe58155471ff9889309936d362a18d479c410171d" exitCode=0
Oct 31 01:04:38.891360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.891283    5419 generic.go:356] "Generic (PLEG): container finished" podID="2b44b15da499055b285bb46b1575da8d" containerID="84aad878c01028b90058c812ce1068a187282153e91e034ee1a8eca77e8ef9e5" exitCode=0
Oct 31 01:04:38.900810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.900780    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a" exitCode=0
Oct 31 01:04:38.901267 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.901246    5419 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-164.us-west-1.compute.internal\" not found"
Oct 31 01:04:38.903318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.903293    5419 generic.go:356] "Generic (PLEG): container finished" podID="43cb7dd9-ef71-43ac-8631-84d0ee80468a" containerID="34573608fc6a60d43c7d6c730336ba52009a7223f07cc550fc3497a49720fbf1" exitCode=0
Oct 31 01:04:38.929702 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.929672    5419 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 01:04:38.930407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.930389    5419 manager.go:324] Recovery completed
Oct 31 01:04:38.955228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.955192    5419 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:04:38.957203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.957178    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:04:38.957327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.957220    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:04:38.957327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.957233    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:04:38.960639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960621    5419 cpu_manager.go:226] "Starting" policy="none"
Oct 31 01:04:38.960639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960639    5419 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960657    5419 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960630    5419 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960790    5419 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960801    5419 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960817    5419 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960822    5419 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 01:04:38.960827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960829    5419 policy_none.go:50] "Start"
Oct 31 01:04:38.961110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960853    5419 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 01:04:38.961110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960863    5419 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 01:04:38.961110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.960998    5419 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 01:04:38.961110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.961007    5419 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 01:04:38.961110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.961012    5419 policy_none.go:44] "Start"
Oct 31 01:04:38.961728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.961704    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:04:38.961843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.961745    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:04:38.961843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.961763    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:04:38.966666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.966621    5419 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 01:04:38.970990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.970972    5419 manager.go:341] "Starting Device Plugin manager"
Oct 31 01:04:38.971100 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.971003    5419 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 01:04:38.971100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971015    5419 server.go:87] "Starting device plugin registration server"
Oct 31 01:04:38.971305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971289    5419 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 01:04:38.971305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971303    5419 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 01:04:38.971491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971442    5419 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 01:04:38.971568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971531    5419 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 01:04:38.971568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971569    5419 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 01:04:38.971766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971575    5419 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 01:04:38.972584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.971655    5419 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:04:38.97155952Z","UUID":"302a039d-f42b-4ce1-8565-fd7b5894357d","Handler":null,"Name":"","Endpoint":""}
Oct 31 01:04:38.974276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.974259    5419 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 01:04:38.974385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.974301    5419 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 01:04:38.975193 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.975163    5419 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 01:04:38.975275 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:38.975224    5419 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-164.us-west-1.compute.internal\" not found"
Oct 31 01:04:38.988575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.988556    5419 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 01:04:38.989954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.989929    5419 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:38.990051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.989982    5419 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:38.990051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:38.990006    5419 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:39.071700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.071669    5419 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 01:04:39.072833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.072812    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 01:04:39.072969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.072849    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 01:04:39.072969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.072864    5419 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-164.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 01:04:39.072969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.072948    5419 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.087848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.087815    5419 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 01:04:39.130457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.130406    5419 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"]
Oct 31 01:04:39.131182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131158    5419 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.131320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131186    5419 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.131320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131176    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerStarted","Data":"a2a92ccc907e89528c6d52ca5bb319a92afc4fcd7cb2ff3ce83ec3c0ed02dd5d"}
Oct 31 01:04:39.131320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131240    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerDied","Data":"84aad878c01028b90058c812ce1068a187282153e91e034ee1a8eca77e8ef9e5"}
Oct 31 01:04:39.131320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131262    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"2b44b15da499055b285bb46b1575da8d","Type":"ContainerStarted","Data":"7fdaa15766bf63bf21bf61bdbae3c9c3046648f4d6ef8d77a8b770c241207589"}
Oct 31 01:04:39.131320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131275    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"7360d5bb713e28adcb32faa1b6003ade","Type":"ContainerStarted","Data":"d55e9063cb67474b8020c9ae7ac7f76195b6e7ef99fe055c20d9eaafaab64a74"}
Oct 31 01:04:39.131320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.131288    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" event={"ID":"7360d5bb713e28adcb32faa1b6003ade","Type":"ContainerStarted","Data":"5a14b75e581f39840dbb044cc47fe708c02401c82cade5fe8cbca9f2193429af"}
Oct 31 01:04:39.189749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.189675    5419 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.189749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.189750    5419 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.200328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.200290    5419 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:39.200485 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:39.200344    5419 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.203494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.203460    5419 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:39.203640 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:39.203517    5419 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.290250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.290203    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.290438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.290266    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.290438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.290319    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7360d5bb713e28adcb32faa1b6003ade-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"7360d5bb713e28adcb32faa1b6003ade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.390682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.390645    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7360d5bb713e28adcb32faa1b6003ade-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"7360d5bb713e28adcb32faa1b6003ade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.390881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.390690    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.390881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.390724    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.390881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.390778    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.390881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.390826    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7360d5bb713e28adcb32faa1b6003ade-config\") pod \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"7360d5bb713e28adcb32faa1b6003ade\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.390881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.390851    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b44b15da499055b285bb46b1575da8d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" (UID: \"2b44b15da499055b285bb46b1575da8d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.775378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.775349    5419 apiserver.go:52] "Watching apiserver"
Oct 31 01:04:39.880200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.880169    5419 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 01:04:39.880763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.880737    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-cmxkm","openshift-insights/insights-runtime-extractor-btnvg","openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz","openshift-network-operator/iptables-alerter-kppxf","kube-system/konnectivity-agent-d9bhz","openshift-ovn-kubernetes/ovnkube-node-fpzqj","openshift-monitoring/node-exporter-fbjh9","openshift-console/downloads-6dd789c8b7-9p7q6","openshift-dns/node-resolver-qctlt","openshift-dns/dns-default-zx8f4","openshift-network-diagnostics/network-check-target-7znmf","kube-system/global-pull-secret-syncer-tmx2h","openshift-monitoring/telemeter-client-757fd9cbf7-46qpv","openshift-multus/network-metrics-daemon-vrg94","openshift-cluster-node-tuning-operator/tuned-wkfwh","openshift-console/console-57878cf6bd-2r9fz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-additional-cni-plugins-b96bx","kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal","openshift-multus/multus-b5mnh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf","openshift-ingress-canary/ingress-canary-dnvvs"]
Oct 31 01:04:39.911796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.911763    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.932621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.932600    5419 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.932754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.932739    5419 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.936231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936187    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 01:04:39.936231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936209    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 01:04:39.936396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936268    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 01:04:39.936396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936209    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 01:04:39.936396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936209    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 01:04:39.936550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936408    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 01:04:39.936550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936419    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 01:04:39.936550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936508    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qlxzn\""
Oct 31 01:04:39.936550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936515    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8rwd\""
Oct 31 01:04:39.936550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936537    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 01:04:39.936550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936538    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936578    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936585    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-55dd6\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936590    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rnpg9\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936603    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-42fft\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936665    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zzhf8\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936712    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936734    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936752    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936769    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 01:04:39.936836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936777    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 01:04:39.937285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.936955    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:04:39.937285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937028    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 01:04:39.937285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937208    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 01:04:39.937285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937250    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lc659\""
Oct 31 01:04:39.937285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937251    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 01:04:39.937285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937282    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 01:04:39.937458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937336    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 01:04:39.937458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937362    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 01:04:39.937939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.937925    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-grw5x\""
Oct 31 01:04:39.990585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.990560    5419 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:39.990727 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:39.990617    5419 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.990858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.990842    5419 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 01:04:39.990899 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:39.990883    5419 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-164.us-west-1.compute.internal"
Oct 31 01:04:39.995063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995041    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-kubelet\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.995165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995081    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-cni-binary-copy\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.995165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995099    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-metrics-client-ca\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.995239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995221    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:39.995271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995244    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-systemd\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.995271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995259    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-hostroot\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.995350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995284    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de59f004-1680-492e-88f9-3278ed575e50-serviceca\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:39.995350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995308    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-lib-modules\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.995447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995375    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-cni-binary-copy\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.995447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995379    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-system-cni-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.995447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995411    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-systemd-units\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.995447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995440    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/198bdfe6-2b3f-4426-a0cc-6568d22b139d-cert\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:39.995573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995466    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-cnibin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.995573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995498    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-trusted-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:39.995573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995559    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-conf\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.995697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995628    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-var-lib-kubelet\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.995697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995651    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:39.995697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995675    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.995697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995695    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysconfig\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995726    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-env-overrides\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995747    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w98f\" (UniqueName: \"kubernetes.io/projected/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-kube-api-access-9w98f\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995763    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8d4p\" (UniqueName: \"kubernetes.io/projected/96abcc08-0354-41d9-baec-af53dd804fdd-kube-api-access-t8d4p\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995790    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-sys-fs\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995805    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-systemd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995825    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.995854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995841    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-netd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995859    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mtd\" (UniqueName: \"kubernetes.io/projected/9cae5eb3-8cac-465d-9b07-f596954cf636-kube-api-access-96mtd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995896    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-dbus\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995930    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995954    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-tmp\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995975    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-env-overrides\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.995982    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cae5eb3-8cac-465d-9b07-f596954cf636-ovn-node-metrics-cert\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996046    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-tmp\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996055    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-script-lib\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996098    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de59f004-1680-492e-88f9-3278ed575e50-host\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996119    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996178    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5v92\" (UniqueName: \"kubernetes.io/projected/120efa71-c477-4936-b599-e9f81f035a25-kube-api-access-b5v92\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.996219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996215    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0e6596-aec8-43f1-9aac-2497c9563bef-crio-socket\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996229    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cae5eb3-8cac-465d-9b07-f596954cf636-ovn-node-metrics-cert\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996319    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-script-lib\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996341    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cfp\" (UniqueName: \"kubernetes.io/projected/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-api-access-78cfp\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996466    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtv7v\" (UniqueName: \"kubernetes.io/projected/198bdfe6-2b3f-4426-a0cc-6568d22b139d-kube-api-access-qtv7v\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996500    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96abcc08-0354-41d9-baec-af53dd804fdd-config-volume\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996530    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996591    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-sys\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.996645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996629    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996698    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996714    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-socket-dir-parent\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996730    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-serving-certs-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996757    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-metrics-client-ca\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996783    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9dd849-9bfc-44bf-8eba-b8a08677c824-konnectivity-ca\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996868    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996886    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-node-log\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.996908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996901    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.997176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996916    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-iptables-alerter-script\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:39.997176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996944    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9dd849-9bfc-44bf-8eba-b8a08677c824-konnectivity-ca\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:04:39.997176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.996948    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-system-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.997176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997113    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv56d\" (UniqueName: \"kubernetes.io/projected/8e7d6f01-4cbb-4865-931e-86cf217018fe-kube-api-access-jv56d\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:39.997176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997161    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997178    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/111a48d2-3e5c-4c68-8215-98b0cfc5b301-tmp\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997198    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997211    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-iptables-alerter-script\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997217    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997255    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997283    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-wtmp\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997291    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/111a48d2-3e5c-4c68-8215-98b0cfc5b301-tmp\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997311    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997338    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-log-socket\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997366    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-conf-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997398    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.997420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997415    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997440    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-slash\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997481    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-config\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997503    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997523    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997540    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-netns\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997557    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkj5v\" (UniqueName: \"kubernetes.io/projected/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-kube-api-access-mkj5v\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997660    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfbg\" (UniqueName: \"kubernetes.io/projected/de59f004-1680-492e-88f9-3278ed575e50-kube-api-access-lkfbg\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997678    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-run\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997693    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96abcc08-0354-41d9-baec-af53dd804fdd-tmp-dir\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997712    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-kubelet\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997735    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997757    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997776    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-tmp-dir\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997790    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st64n\" (UniqueName: \"kubernetes.io/projected/b865ce7c-d25a-4154-a2cd-3136b2937143-kube-api-access-st64n\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997797    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cae5eb3-8cac-465d-9b07-f596954cf636-ovnkube-config\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997804    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997840    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.998002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.997877    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96abcc08-0354-41d9-baec-af53dd804fdd-tmp-dir\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998172    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-original-pull-secret\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998237    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-bin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998245    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-tmp-dir\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998271    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-multus-certs\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998320    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998375    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998413    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-ovn\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998438    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998463    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8h46\" (UniqueName: \"kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998487    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-bin\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998511    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71865382-46e6-4f6a-bb1f-9b5406439186-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998561    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-tls\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998592    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998622    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-device-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998640    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998658    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-host-slash\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998674    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-textfile\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998689    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65jc4\" (UniqueName: \"kubernetes.io/projected/43cb7dd9-ef71-43ac-8631-84d0ee80468a-kube-api-access-65jc4\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998733    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998749    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-modprobe-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998767    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998775    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-textfile\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998783    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998821    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96abcc08-0354-41d9-baec-af53dd804fdd-metrics-tls\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998849    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-sys\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:39.998927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998879    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-root\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998946    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-host\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998967    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-var-lib-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.998984    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-netns\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999025    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-etc-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999046    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-registration-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999088    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999120    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngvl\" (UniqueName: \"kubernetes.io/projected/111a48d2-3e5c-4c68-8215-98b0cfc5b301-kube-api-access-tngvl\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999186    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-kubernetes\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999215    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-multus\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999242    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94xd\" (UniqueName: \"kubernetes.io/projected/549daf04-71a7-40cd-970b-0d14584c26a4-kube-api-access-x94xd\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999269    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4vd4\" (UniqueName: \"kubernetes.io/projected/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-kube-api-access-g4vd4\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999319    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-kubelet-config\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999347    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzj7\" (UniqueName: \"kubernetes.io/projected/71865382-46e6-4f6a-bb1f-9b5406439186-kube-api-access-9nzj7\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999374    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999402    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-k8s-cni-cncf-io\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999429    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999460    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999504    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w99sz\" (UniqueName: \"kubernetes.io/projected/221a9c21-35c6-4628-87ca-0cc77c4b3595-kube-api-access-w99sz\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999533    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0e6596-aec8-43f1-9aac-2497c9563bef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999634    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-hosts-file\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999662    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cnibin\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999697    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999726    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-socket-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999752    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999778    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-etc-kubernetes\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999807    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999834    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999861    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999896    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999926    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-federate-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999953    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9dd849-9bfc-44bf-8eba-b8a08677c824-agent-certs\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:39.999981    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.000008    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-multus-daemon-config\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.000041    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0e6596-aec8-43f1-9aac-2497c9563bef-data-volume\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.000223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.000066    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-tuned\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.001773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.000285    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.001773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.000346    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-tuned\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.001773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.001114    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.001773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.001471    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af0e6596-aec8-43f1-9aac-2497c9563bef-data-volume\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.001773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.001555    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.001773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.001576    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9dd849-9bfc-44bf-8eba-b8a08677c824-agent-certs\") pod \"konnectivity-agent-d9bhz\" (UID: \"1c9dd849-9bfc-44bf-8eba-b8a08677c824\") " pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:04:40.002039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.001795    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/120efa71-c477-4936-b599-e9f81f035a25-multus-daemon-config\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.004667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.003120    5419 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 01:04:40.035154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035070    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 01:04:40.035154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035093    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 01:04:40.035323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035194    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ln2ts\""
Oct 31 01:04:40.035323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035200    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 01:04:40.035323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035214    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xgzbp\""
Oct 31 01:04:40.035575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035551    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 01:04:40.036116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.035981    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 01:04:40.036116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036034    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.036116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036043    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 01:04:40.036116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036039    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 01:04:40.036116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036064    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cm644\""
Oct 31 01:04:40.036116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036094    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 01:04:40.036411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036129    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 01:04:40.036411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036197    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 01:04:40.036411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036291    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 01:04:40.036529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036484    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 01:04:40.036582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036536    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ch46h\""
Oct 31 01:04:40.036582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036544    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 01:04:40.036582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036538    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 01:04:40.036725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036610    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 01:04:40.036725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036632    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvb7t\""
Oct 31 01:04:40.036725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036639    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.036725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036655    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.036725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036689    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 01:04:40.036725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036711    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.036950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036770    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 01:04:40.036950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036803    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 01:04:40.037024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036954    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.037024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.036959    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.037287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037272    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.037414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037397    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.037496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037476    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.037549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037524    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.037549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037542    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.037637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037550    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.037637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037568    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7pd7l\""
Oct 31 01:04:40.037637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037586    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.037637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037596    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 01:04:40.037637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037604    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037675    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037750    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037796    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037843    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037849    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037858    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037870    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037870    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h9z86\""
Oct 31 01:04:40.037906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.037885    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038010    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038025    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038038    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038069    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6cbrt\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038071    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038073    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038068    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038070    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038106    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038106    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038119    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038115    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5sjdn\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038186    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 01:04:40.038293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038231    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 01:04:40.038817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038425    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4fbhm\""
Oct 31 01:04:40.038817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038492    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7g746\""
Oct 31 01:04:40.038817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038525    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kk499\""
Oct 31 01:04:40.038817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038545    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.038817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.038560    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tg5xh\""
Oct 31 01:04:40.039484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039436    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.039637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039500    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71865382-46e6-4f6a-bb1f-9b5406439186-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:40.039637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039518    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.039637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039527    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.039637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039540    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96abcc08-0354-41d9-baec-af53dd804fdd-metrics-tls\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:40.039637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039533    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-tls\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.039637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.039554    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:40.041293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.041276    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549daf04-71a7-40cd-970b-0d14584c26a4-metrics-certs\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:40.041373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.041315    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af0e6596-aec8-43f1-9aac-2497c9563bef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.041432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.041420    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.041489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.041456    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.041489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.041463    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.041610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.041597    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-federate-client-tls\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.042380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.042360    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-accelerators-collector-config\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.045577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.045557    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b865ce7c-d25a-4154-a2cd-3136b2937143-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.045726 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.045693    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43cb7dd9-ef71-43ac-8631-84d0ee80468a-metrics-client-ca\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.045772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.045747    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/198bdfe6-2b3f-4426-a0cc-6568d22b139d-cert\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:40.045886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.045868    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de59f004-1680-492e-88f9-3278ed575e50-serviceca\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:40.047075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.047061    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96abcc08-0354-41d9-baec-af53dd804fdd-config-volume\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:40.047123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.047080    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-metrics-client-ca\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.047123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.047098    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-serving-certs-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.048271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.048248    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71865382-46e6-4f6a-bb1f-9b5406439186-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:40.087907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.087870    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 01:04:40.088058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.087945    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 01:04:40.088411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088386    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 01:04:40.088485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088408    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv56d\" (UniqueName: \"kubernetes.io/projected/8e7d6f01-4cbb-4865-931e-86cf217018fe-kube-api-access-jv56d\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.088556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088527    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cfp\" (UniqueName: \"kubernetes.io/projected/af0e6596-aec8-43f1-9aac-2497c9563bef-kube-api-access-78cfp\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.088617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088564    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w98f\" (UniqueName: \"kubernetes.io/projected/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-kube-api-access-9w98f\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:40.088678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088647    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5v92\" (UniqueName: \"kubernetes.io/projected/120efa71-c477-4936-b599-e9f81f035a25-kube-api-access-b5v92\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088803    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mtd\" (UniqueName: \"kubernetes.io/projected/9cae5eb3-8cac-465d-9b07-f596954cf636-kube-api-access-96mtd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088909    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st64n\" (UniqueName: \"kubernetes.io/projected/b865ce7c-d25a-4154-a2cd-3136b2937143-kube-api-access-st64n\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.088944    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8d4p\" (UniqueName: \"kubernetes.io/projected/96abcc08-0354-41d9-baec-af53dd804fdd-kube-api-access-t8d4p\") pod \"dns-default-zx8f4\" (UID: \"96abcc08-0354-41d9-baec-af53dd804fdd\") " pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089253    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngvl\" (UniqueName: \"kubernetes.io/projected/111a48d2-3e5c-4c68-8215-98b0cfc5b301-kube-api-access-tngvl\") pod \"downloads-6dd789c8b7-9p7q6\" (UID: \"111a48d2-3e5c-4c68-8215-98b0cfc5b301\") " pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089256    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65jc4\" (UniqueName: \"kubernetes.io/projected/43cb7dd9-ef71-43ac-8631-84d0ee80468a-kube-api-access-65jc4\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089444    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtv7v\" (UniqueName: \"kubernetes.io/projected/198bdfe6-2b3f-4426-a0cc-6568d22b139d-kube-api-access-qtv7v\") pod \"ingress-canary-dnvvs\" (UID: \"198bdfe6-2b3f-4426-a0cc-6568d22b139d\") " pod="openshift-ingress-canary/ingress-canary-dnvvs"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089452    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkj5v\" (UniqueName: \"kubernetes.io/projected/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-kube-api-access-mkj5v\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089482    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzj7\" (UniqueName: \"kubernetes.io/projected/71865382-46e6-4f6a-bb1f-9b5406439186-kube-api-access-9nzj7\") pod \"prometheus-operator-9cd89b74c-w4xxz\" (UID: \"71865382-46e6-4f6a-bb1f-9b5406439186\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-w4xxz"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089576    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4vd4\" (UniqueName: \"kubernetes.io/projected/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-kube-api-access-g4vd4\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089736    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089784    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w99sz\" (UniqueName: \"kubernetes.io/projected/221a9c21-35c6-4628-87ca-0cc77c4b3595-kube-api-access-w99sz\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.089839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089802    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94xd\" (UniqueName: \"kubernetes.io/projected/549daf04-71a7-40cd-970b-0d14584c26a4-kube-api-access-x94xd\") pod \"network-metrics-daemon-vrg94\" (UID: \"549daf04-71a7-40cd-970b-0d14584c26a4\") " pod="openshift-multus/network-metrics-daemon-vrg94"
Oct 31 01:04:40.090376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.089855    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfbg\" (UniqueName: \"kubernetes.io/projected/de59f004-1680-492e-88f9-3278ed575e50-kube-api-access-lkfbg\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:40.090376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.090053    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbt4j\" (UniqueName: \"kubernetes.io/projected/2fa45a8e-78a1-4cce-96e8-5ba324073c61-kube-api-access-nbt4j\") pod \"network-check-target-7znmf\" (UID: \"2fa45a8e-78a1-4cce-96e8-5ba324073c61\") " pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:40.091858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.091841    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:04:40.096040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.096016    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b865ce7c-d25a-4154-a2cd-3136b2937143-telemeter-trusted-ca-bundle\") pod \"telemeter-client-757fd9cbf7-46qpv\" (UID: \"b865ce7c-d25a-4154-a2cd-3136b2937143\") " pod="openshift-monitoring/telemeter-client-757fd9cbf7-46qpv"
Oct 31 01:04:40.101233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101215    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-systemd-units\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101241    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-cnibin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.101307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101258    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-conf\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.101307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101273    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-var-lib-kubelet\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.101307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101288    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101312    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101328    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-cnibin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101355    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-var-lib-kubelet\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101332    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-systemd-units\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101335    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysconfig\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101376    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysconfig\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101369    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101399    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-sys-fs\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101415    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-systemd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101423    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-conf\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101430    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101447    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101468    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-netd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101478    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-sys-fs\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101471    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-systemd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101497    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-netd\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.101495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101493    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-dbus\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101528    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de59f004-1680-492e-88f9-3278ed575e50-host\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101541    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-dbus\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101552    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101570    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de59f004-1680-492e-88f9-3278ed575e50-host\") pod \"node-ca-cmxkm\" (UID: \"de59f004-1680-492e-88f9-3278ed575e50\") " pod="openshift-image-registry/node-ca-cmxkm"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101576    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0e6596-aec8-43f1-9aac-2497c9563bef-crio-socket\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101589    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101610    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af0e6596-aec8-43f1-9aac-2497c9563bef-crio-socket\") pod \"insights-runtime-extractor-btnvg\" (UID: \"af0e6596-aec8-43f1-9aac-2497c9563bef\") " pod="openshift-insights/insights-runtime-extractor-btnvg"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101610    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101640    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-sys\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101659    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101673    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-socket-dir-parent\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101686    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-sys\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101691    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-node-log\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101710    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-node-log\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101711    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101734    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-system-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101737    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101763    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-wtmp\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101771    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101781    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-socket-dir-parent\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101788    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-log-socket\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101790    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-system-cni-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101816    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-node-exporter-wtmp\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101817    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-conf-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101842    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101832    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-log-socket\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101872    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-slash\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101892    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-netns\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101891    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-multus-conf-dir\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101899    5419 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101907    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-run\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101937    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-netns\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101936    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-slash\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101939    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-run\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101960    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-kubelet\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101983    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-kubelet\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.101988    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-bin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102012    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-multus-certs\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102021    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102037    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102047    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-multus-certs\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102053    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-bin\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-ovn\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102115    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8h46\" (UniqueName: \"kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102161    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-bin\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102171    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-run-ovn\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102196    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102211    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-cni-bin\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102218    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-device-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102251    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-host-slash\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102273    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102293    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-modprobe-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102316    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-sys\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102330    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda-host-slash\") pod \"iptables-alerter-kppxf\" (UID: \"bbe8ed9e-59cf-4fc6-a568-aa8d71d82bda\") " pod="openshift-network-operator/iptables-alerter-kppxf"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102334    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-sysctl-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102338    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-root\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102368    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-root\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102373    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-host\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102382    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-device-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102397    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-var-lib-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102421    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-netns\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.102375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102422    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102444    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-etc-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102475    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-registration-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102479    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-host\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102487    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-run-netns\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102422    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43cb7dd9-ef71-43ac-8631-84d0ee80468a-sys\") pod \"node-exporter-fbjh9\" (UID: \"43cb7dd9-ef71-43ac-8631-84d0ee80468a\") " pod="openshift-monitoring/node-exporter-fbjh9"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102499    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-kubernetes\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102481    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-modprobe-d\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102451    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-var-lib-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102522    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-etc-openvswitch\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102536    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-multus\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102543    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-kubernetes\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102565    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-kubelet-config\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102591    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-registration-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102607    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-k8s-cni-cncf-io\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102634    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102639    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-cni-multus\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102640    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-run-k8s-cni-cncf-io\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102660    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-hosts-file\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102640    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ef840aa5-db4c-4422-a2d4-702a5c8bdcb0-kubelet-config\") pod \"global-pull-secret-syncer-tmx2h\" (UID: \"ef840aa5-db4c-4422-a2d4-702a5c8bdcb0\") " pod="kube-system/global-pull-secret-syncer-tmx2h"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102684    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cnibin\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102717    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aa5bbf79-871a-4f48-beea-cb3bf34fbe64-hosts-file\") pod \"node-resolver-qctlt\" (UID: \"aa5bbf79-871a-4f48-beea-cb3bf34fbe64\") " pod="openshift-dns/node-resolver-qctlt"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102710    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-socket-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102746    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-cnibin\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102755    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-etc-kubernetes\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102770    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102786    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102801    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d6f01-4cbb-4865-931e-86cf217018fe-socket-dir\") pod \"aws-ebs-csi-driver-node-mnksf\" (UID: \"8e7d6f01-4cbb-4865-931e-86cf217018fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mnksf"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102816    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102840    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-kubelet\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102867    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-systemd\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102870    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102888    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-hostroot\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102910    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-etc-kubernetes\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102916    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-lib-modules\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102940    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-system-cni-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.102955    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-host-var-lib-kubelet\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103017    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cae5eb3-8cac-465d-9b07-f596954cf636-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fpzqj\" (UID: \"9cae5eb3-8cac-465d-9b07-f596954cf636\") " pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103037    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ee4c566-ef1d-42ec-8e17-3de154e4ec69-system-cni-dir\") pod \"multus-additional-cni-plugins-b96bx\" (UID: \"6ee4c566-ef1d-42ec-8e17-3de154e4ec69\") " pod="openshift-multus/multus-additional-cni-plugins-b96bx"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103059    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-etc-systemd\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103071    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/120efa71-c477-4936-b599-e9f81f035a25-hostroot\") pod \"multus-b5mnh\" (UID: \"120efa71-c477-4936-b599-e9f81f035a25\") " pod="openshift-multus/multus-b5mnh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103100    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103166    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/221a9c21-35c6-4628-87ca-0cc77c4b3595-lib-modules\") pod \"tuned-wkfwh\" (UID: \"221a9c21-35c6-4628-87ca-0cc77c4b3595\") " pod="openshift-cluster-node-tuning-operator/tuned-wkfwh"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103461    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.104540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.103731    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.105885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.104758    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.105885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.104842    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.167522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.167496    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8h46\" (UniqueName: \"kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46\") pod \"console-57878cf6bd-2r9fz\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") " pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.192210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.192129    5419 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:04:40.232639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.232599    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:40.506099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.506070    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zx8f4"
Oct 31 01:04:40.604090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.604005    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57878cf6bd-2r9fz"]
Oct 31 01:04:40.607723 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:04:40.607697    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf03a56d_1c3f_486b_bb48_df377b3d4768.slice/crio-569d1a90377a1fe92f30145ed98b1e289e9a6acdeed9d58cddeaa1cbad9dc18c WatchSource:0}: Error finding container 569d1a90377a1fe92f30145ed98b1e289e9a6acdeed9d58cddeaa1cbad9dc18c: Status 404 returned error can't find the container with id 569d1a90377a1fe92f30145ed98b1e289e9a6acdeed9d58cddeaa1cbad9dc18c
Oct 31 01:04:40.934396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:40.934319    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57878cf6bd-2r9fz" event={"ID":"bf03a56d-1c3f-486b-bb48-df377b3d4768","Type":"ContainerStarted","Data":"569d1a90377a1fe92f30145ed98b1e289e9a6acdeed9d58cddeaa1cbad9dc18c"}
Oct 31 01:04:41.265848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:41.265814    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:41.941960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:41.941926    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9p7q6" event={"ID":"111a48d2-3e5c-4c68-8215-98b0cfc5b301","Type":"ContainerStarted","Data":"06abbb24b2b37a8490f47d7e2b3436aba48d4ba2665f7ac7c7f9287dabdf565d"}
Oct 31 01:04:42.675581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:42.675541    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:42.687602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:42.687559    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-9p7q6"
Oct 31 01:04:42.796863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:42.796825    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:04:42.797126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:42.797109    5419 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 01:04:44.586858 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:04:44.586825    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:04:44.957243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:44.957117    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57878cf6bd-2r9fz" event={"ID":"bf03a56d-1c3f-486b-bb48-df377b3d4768","Type":"ContainerStarted","Data":"26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738"}
Oct 31 01:04:45.074083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:45.074016    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57878cf6bd-2r9fz" podStartSLOduration=2.043253876 podStartE2EDuration="6.073996955s" totalImagesPullingTime="4.030743079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:45.073772608 +0000 UTC m=+6.374729227" watchObservedRunningTime="2026-10-31 01:04:45.073996955 +0000 UTC m=+6.374953531"
Oct 31 01:04:48.580678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:48.580583    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:04:50.522057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:50.522023    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-d9bhz"
Oct 31 01:04:51.595062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:51.595019    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:52.465708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:52.465670    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:52.470583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:52.470555    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:04:52.990123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:04:52.990096    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:05:02.694973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:02.694938    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fpzqj"
Oct 31 01:05:05.217980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.217947    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:05:05.218558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.218516    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="alertmanager" containerID="cri-o://3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449" gracePeriod=120
Oct 31 01:05:05.218558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.218547    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="config-reloader" containerID="cri-o://40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1" gracePeriod=120
Oct 31 01:05:05.218763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.218544    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy" containerID="cri-o://888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584" gracePeriod=120
Oct 31 01:05:05.218763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.218586    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-web" containerID="cri-o://2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99" gracePeriod=120
Oct 31 01:05:05.218763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.218537    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-metric" containerID="cri-o://43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293" gracePeriod=120
Oct 31 01:05:05.218763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.218618    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="prom-label-proxy" containerID="cri-o://3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0" gracePeriod=120
Oct 31 01:05:05.515818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.515788    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:05.606425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606394    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606435    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606461    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606495    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606697    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606771    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606815    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.606855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606853    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.607089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606882    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:05.607089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606902    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.607089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606934    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.607089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.606982    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.609085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.607960    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:05.609085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.608217    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.609085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.608266    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") pod \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\" (UID: \"c1d07da0-09ea-4f37-a5e4-8690f2c3882f\") "
Oct 31 01:05:05.609085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.608521    5419 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-metrics-client-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.609085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.608543    5419 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.609085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.608856    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.611170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611125    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.611285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611248    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:05.611409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611374    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.611522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611464    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.611701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611684    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:05.611775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611753    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.611860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.611836    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.612060 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.612045    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "kube-api-access-pjf75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:05.614969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.614947    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.620267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.620243    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config" pod "c1d07da0-09ea-4f37-a5e4-8690f2c3882f" (UID: "c1d07da0-09ea-4f37-a5e4-8690f2c3882f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:05.709650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709618    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjf75\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-kube-api-access-pjf75\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709647    5419 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-alertmanager-main-db\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709659    5419 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-out\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709670    5419 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709680    5419 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-web-config\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709688    5419 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709697    5419 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-main-tls\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709707    5419 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-cluster-tls-config\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709715    5419 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709723    5419 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-tls-assets\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.709836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:05.709734    5419 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1d07da0-09ea-4f37-a5e4-8690f2c3882f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:06.032250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032220    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0" exitCode=0
Oct 31 01:05:06.032250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032244    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293" exitCode=0
Oct 31 01:05:06.032250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032251    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584" exitCode=0
Oct 31 01:05:06.032250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032256    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99" exitCode=0
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032262    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1" exitCode=0
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032267    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449" exitCode=0
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032301    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032313    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"}
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032390    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"}
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032421    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"}
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032445    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"}
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032454    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032477    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"}
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032503    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"}
Oct 31 01:05:06.032547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.032532    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1d07da0-09ea-4f37-a5e4-8690f2c3882f","Type":"ContainerDied","Data":"7525d3711c66a069e39bba425d72b01a3e506f4cb35796a8d03e287dd1eafd05"}
Oct 31 01:05:06.040049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.040021    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.046738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.046722    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.053403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.053382    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.060727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.060429    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.067322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.067302    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.074154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.074116    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.080671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.080654    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.080941 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.080922    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.080989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.080952    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"} err="failed to get container status \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist"
Oct 31 01:05:06.081035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.080993    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.081240 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.081225    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.081289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081246    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"} err="failed to get container status \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist"
Oct 31 01:05:06.081289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081260    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.081511 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.081483    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.081559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081505    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"} err="failed to get container status \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist"
Oct 31 01:05:06.081559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081518    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.081726 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.081711    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.081769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081729    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"} err="failed to get container status \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist"
Oct 31 01:05:06.081769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081742    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.081938 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.081924    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.081974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081942    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"} err="failed to get container status \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist"
Oct 31 01:05:06.081974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.081954    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.082160 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.082130    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.082221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082163    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"} err="failed to get container status \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist"
Oct 31 01:05:06.082221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082175    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.082571 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:06.082556    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.082619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082574    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"} err="failed to get container status \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist"
Oct 31 01:05:06.082619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082586    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.082775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082760    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"} err="failed to get container status \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist"
Oct 31 01:05:06.082817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082775    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.082968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082950    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"} err="failed to get container status \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist"
Oct 31 01:05:06.083016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.082968    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.083184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083161    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"} err="failed to get container status \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist"
Oct 31 01:05:06.083184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083182    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.083440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083419    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"} err="failed to get container status \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist"
Oct 31 01:05:06.083480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083441    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.083682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083666    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"} err="failed to get container status \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist"
Oct 31 01:05:06.083727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083682    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.083911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083893    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"} err="failed to get container status \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist"
Oct 31 01:05:06.083952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.083912    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.084148 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084120    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"} err="failed to get container status \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist"
Oct 31 01:05:06.084193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084159    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.084381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084367    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"} err="failed to get container status \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist"
Oct 31 01:05:06.084426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084381    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.084602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084584    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"} err="failed to get container status \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist"
Oct 31 01:05:06.084642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084603    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.084820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084802    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"} err="failed to get container status \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist"
Oct 31 01:05:06.084856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.084819    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.085020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085004    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"} err="failed to get container status \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist"
Oct 31 01:05:06.085058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085022    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.085304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085288    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"} err="failed to get container status \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist"
Oct 31 01:05:06.085304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085302    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.085523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085505    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"} err="failed to get container status \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist"
Oct 31 01:05:06.085564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085524    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.085766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085748    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"} err="failed to get container status \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist"
Oct 31 01:05:06.085814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085766    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.086005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.085988    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"} err="failed to get container status \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist"
Oct 31 01:05:06.086107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086005    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.086264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086247    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"} err="failed to get container status \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist"
Oct 31 01:05:06.086317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086265    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.086477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086460    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"} err="failed to get container status \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist"
Oct 31 01:05:06.086526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086477    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.086703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086683    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"} err="failed to get container status \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist"
Oct 31 01:05:06.086748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086704    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.086930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086915    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"} err="failed to get container status \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist"
Oct 31 01:05:06.086982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.086930    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.087132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087116    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"} err="failed to get container status \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist"
Oct 31 01:05:06.087187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087131    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.087344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087329    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"} err="failed to get container status \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist"
Oct 31 01:05:06.087384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087344    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.087544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087526    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"} err="failed to get container status \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist"
Oct 31 01:05:06.087587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087544    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.087735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087719    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"} err="failed to get container status \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist"
Oct 31 01:05:06.087777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087734    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.087933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087918    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"} err="failed to get container status \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist"
Oct 31 01:05:06.087973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.087933    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.088116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088101    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"} err="failed to get container status \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist"
Oct 31 01:05:06.088171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088116    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.088311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088293    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"} err="failed to get container status \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist"
Oct 31 01:05:06.088348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088311    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.088497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088482    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"} err="failed to get container status \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist"
Oct 31 01:05:06.088534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088498    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.088680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088665    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"} err="failed to get container status \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist"
Oct 31 01:05:06.088721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088680    5419 scope.go:122] "RemoveContainer" containerID="3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"
Oct 31 01:05:06.088868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088854    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0"} err="failed to get container status \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": rpc error: code = NotFound desc = could not find container \"3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0\": container with ID starting with 3cf63ef2f9c6c4fa63da6247a189cbdb6c6eeae1c2b91d8f5f9fcdde5d809dd0 not found: ID does not exist"
Oct 31 01:05:06.088908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.088868    5419 scope.go:122] "RemoveContainer" containerID="43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"
Oct 31 01:05:06.089032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089017    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293"} err="failed to get container status \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": rpc error: code = NotFound desc = could not find container \"43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293\": container with ID starting with 43121a9eebf0299cf9f33ee095aa04dc35df9c3f8a3c9bd16158c38f68fa3293 not found: ID does not exist"
Oct 31 01:05:06.089075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089031    5419 scope.go:122] "RemoveContainer" containerID="888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"
Oct 31 01:05:06.089226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089209    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584"} err="failed to get container status \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": rpc error: code = NotFound desc = could not find container \"888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584\": container with ID starting with 888e480311cf2b4ecabb3f7d14b0f3ce7da670269552328c020f09d74a2a2584 not found: ID does not exist"
Oct 31 01:05:06.089226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089225    5419 scope.go:122] "RemoveContainer" containerID="2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"
Oct 31 01:05:06.089428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089412    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99"} err="failed to get container status \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": rpc error: code = NotFound desc = could not find container \"2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99\": container with ID starting with 2b8596db213ca9145350358fb6a092faea36f90bbfe030d96946afe7b29f7d99 not found: ID does not exist"
Oct 31 01:05:06.089465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089428    5419 scope.go:122] "RemoveContainer" containerID="40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"
Oct 31 01:05:06.089643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089624    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1"} err="failed to get container status \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": rpc error: code = NotFound desc = could not find container \"40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1\": container with ID starting with 40832eb93ab52dc2ac9ffdd5327b1365bbd482556a3e27c0eec5cbfde65e58a1 not found: ID does not exist"
Oct 31 01:05:06.089686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089644    5419 scope.go:122] "RemoveContainer" containerID="3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"
Oct 31 01:05:06.089849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089835    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449"} err="failed to get container status \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": rpc error: code = NotFound desc = could not find container \"3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449\": container with ID starting with 3c04e8b3a9647b4327249b2e80fce21fbb017b527c2cf47d7ec2b8507ea49449 not found: ID does not exist"
Oct 31 01:05:06.089890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.089849    5419 scope.go:122] "RemoveContainer" containerID="deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"
Oct 31 01:05:06.090068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.090051    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a"} err="failed to get container status \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": rpc error: code = NotFound desc = could not find container \"deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a\": container with ID starting with deb63cf53f8b1ebf4063240e2374f29186b95422cd55d71ecf52cce85855eb2a not found: ID does not exist"
Oct 31 01:05:06.199551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.199519    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:05:06.249508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.249476    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:05:06.249884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.249763    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:05:06.250024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250010    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="config-reloader"
Oct 31 01:05:06.250059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250025    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="config-reloader"
Oct 31 01:05:06.250059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250039    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy"
Oct 31 01:05:06.250059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250044    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy"
Oct 31 01:05:06.250059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250052    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="prom-label-proxy"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250058    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="prom-label-proxy"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250068    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-metric"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250076    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-metric"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250086    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="alertmanager"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250090    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="alertmanager"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250096    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="init-config-reloader"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250101    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="init-config-reloader"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250112    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-web"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250116    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-web"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250191    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-web"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250201    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="config-reloader"
Oct 31 01:05:06.250206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250207    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy"
Oct 31 01:05:06.250650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250213    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="prom-label-proxy"
Oct 31 01:05:06.250650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250219    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="kube-rbac-proxy-metric"
Oct 31 01:05:06.250650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250227    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="alertmanager"
Oct 31 01:05:06.250650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.250234    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" containerName="init-config-reloader"
Oct 31 01:05:06.270399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.270378    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.322987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.322920    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 01:05:06.322987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.322922    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 01:05:06.323229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323121    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 01:05:06.323229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323182    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 01:05:06.323229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323217    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 01:05:06.323355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323238    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 01:05:06.323404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323385    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 01:05:06.323444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323398    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 01:05:06.323481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.323461    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7pd7l\""
Oct 31 01:05:06.372411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.372383    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 01:05:06.373279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.373260    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:05:06.416229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416201    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416233    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416263    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd6977b9-1cad-406c-b35f-4da0ba739d03-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416282    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416379    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xn4\" (UniqueName: \"kubernetes.io/projected/cd6977b9-1cad-406c-b35f-4da0ba739d03-kube-api-access-d8xn4\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416406    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-web-config\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416430    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416494    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd6977b9-1cad-406c-b35f-4da0ba739d03-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416523    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd6977b9-1cad-406c-b35f-4da0ba739d03-config-out\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416599    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416620    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6977b9-1cad-406c-b35f-4da0ba739d03-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416641    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd6977b9-1cad-406c-b35f-4da0ba739d03-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.416678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.416664    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517038    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-web-config\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517075    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517105    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd6977b9-1cad-406c-b35f-4da0ba739d03-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517154    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd6977b9-1cad-406c-b35f-4da0ba739d03-config-out\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517188    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517213    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6977b9-1cad-406c-b35f-4da0ba739d03-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517241    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd6977b9-1cad-406c-b35f-4da0ba739d03-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517272    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517307    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517328    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517361    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd6977b9-1cad-406c-b35f-4da0ba739d03-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517386    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517413    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xn4\" (UniqueName: \"kubernetes.io/projected/cd6977b9-1cad-406c-b35f-4da0ba739d03-kube-api-access-d8xn4\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.517712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.517575    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cd6977b9-1cad-406c-b35f-4da0ba739d03-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.518581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.518556    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd6977b9-1cad-406c-b35f-4da0ba739d03-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.518804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.518754    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd6977b9-1cad-406c-b35f-4da0ba739d03-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.520603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.520576    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-web-config\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.520926    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd6977b9-1cad-406c-b35f-4da0ba739d03-config-out\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.521106    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.521124    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd6977b9-1cad-406c-b35f-4da0ba739d03-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.521112    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.521266    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-config-volume\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.521442    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.521997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.521983    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.522639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.522616    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cd6977b9-1cad-406c-b35f-4da0ba739d03-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.575945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.575881    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xn4\" (UniqueName: \"kubernetes.io/projected/cd6977b9-1cad-406c-b35f-4da0ba739d03-kube-api-access-d8xn4\") pod \"alertmanager-main-0\" (UID: \"cd6977b9-1cad-406c-b35f-4da0ba739d03\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.678987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.678952    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 01:05:06.834799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:06.834731    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d07da0-09ea-4f37-a5e4-8690f2c3882f" path="/var/lib/kubelet/pods/c1d07da0-09ea-4f37-a5e4-8690f2c3882f/volumes"
Oct 31 01:05:07.184913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:07.184843    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 01:05:07.187621 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:05:07.187590    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6977b9_1cad_406c_b35f_4da0ba739d03.slice/crio-e8aa9f22ec1ca493cd3a8a953e218abfd51050b0387a02b00d8259e85a5ee74c WatchSource:0}: Error finding container e8aa9f22ec1ca493cd3a8a953e218abfd51050b0387a02b00d8259e85a5ee74c: Status 404 returned error can't find the container with id e8aa9f22ec1ca493cd3a8a953e218abfd51050b0387a02b00d8259e85a5ee74c
Oct 31 01:05:08.046336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:08.046292    5419 generic.go:356] "Generic (PLEG): container finished" podID="cd6977b9-1cad-406c-b35f-4da0ba739d03" containerID="338574b1c505f541161d861a2f515bd2c0862dfd091521f6f28bfc4312cae8c7" exitCode=0
Oct 31 01:05:08.046750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:08.046336    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerDied","Data":"338574b1c505f541161d861a2f515bd2c0862dfd091521f6f28bfc4312cae8c7"}
Oct 31 01:05:08.046750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:08.046423    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"e8aa9f22ec1ca493cd3a8a953e218abfd51050b0387a02b00d8259e85a5ee74c"}
Oct 31 01:05:08.583832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:08.583807    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7znmf"
Oct 31 01:05:09.053883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:09.053849    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"12053b0581268dc6a1ed323c1261ec355fe666dabf239a70e5cded0e0fbf059c"}
Oct 31 01:05:09.054319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:09.053926    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"8269f9c5650b539ee08da694fcabc694c5371bf299b9037b688ca2a75b0b3aa1"}
Oct 31 01:05:09.054319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:09.053968    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"92d93b38588f23dc5aabed4066140aaaaa49b418726b07f2fade963d911f42de"}
Oct 31 01:05:09.054319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:09.054010    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"c065c7f47c791c353f955e2370ae01a97538eccc67a601ca0b492d804a9e478f"}
Oct 31 01:05:09.054319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:09.054044    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"5ae5578975d70a6db5665bdab69b24a8fb990ed9ec42dd3813d34ad626202429"}
Oct 31 01:05:09.054319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:09.054078    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cd6977b9-1cad-406c-b35f-4da0ba739d03","Type":"ContainerStarted","Data":"dbc965f061b4cc92ce7029634fb9c4a80c004ace09d39a06980dbbc813f57dd7"}
Oct 31 01:05:11.299946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:11.299892    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.28708925 podStartE2EDuration="5.299878405s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.789155ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:09.192300551 +0000 UTC m=+30.493257127" watchObservedRunningTime="2026-10-31 01:05:11.299878405 +0000 UTC m=+32.600834983"
Oct 31 01:05:11.300345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:11.300127    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57878cf6bd-2r9fz"]
Oct 31 01:05:14.580202 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:14.580160    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:05:36.319362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.319292    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57878cf6bd-2r9fz" podUID="bf03a56d-1c3f-486b-bb48-df377b3d4768" containerName="console" containerID="cri-o://26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738" gracePeriod=15
Oct 31 01:05:36.573626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.573571    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:05:36.754100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754067    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754120    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754165    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754181    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754196    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754240    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754291    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46\" (UniqueName: \"kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754356    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config\") pod \"bf03a56d-1c3f-486b-bb48-df377b3d4768\" (UID: \"bf03a56d-1c3f-486b-bb48-df377b3d4768\") "
Oct 31 01:05:36.754679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754627    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:36.754865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754681    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:36.754865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754696    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:05:36.754976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754865    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:36.754976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.754899    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:36.756782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.756756    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:36.756782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.756766    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "kube-api-access-t8h46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:36.756896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.756773    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert" pod "bf03a56d-1c3f-486b-bb48-df377b3d4768" (UID: "bf03a56d-1c3f-486b-bb48-df377b3d4768"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:36.855563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855533    5419 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-oauth-config\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855558    5419 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855567    5419 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-oauth-serving-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855577    5419 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-serving-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855586    5419 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-console-config\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855594    5419 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf03a56d-1c3f-486b-bb48-df377b3d4768-trusted-ca-bundle\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855602    5419 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf03a56d-1c3f-486b-bb48-df377b3d4768-tmp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:36.855732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:36.855610    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8h46\" (UniqueName: \"kubernetes.io/projected/bf03a56d-1c3f-486b-bb48-df377b3d4768-kube-api-access-t8h46\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:37.154178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.154078    5419 generic.go:356] "Generic (PLEG): container finished" podID="bf03a56d-1c3f-486b-bb48-df377b3d4768" containerID="26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738" exitCode=0
Oct 31 01:05:37.154178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.154124    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57878cf6bd-2r9fz"
Oct 31 01:05:37.154178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.154169    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57878cf6bd-2r9fz" event={"ID":"bf03a56d-1c3f-486b-bb48-df377b3d4768","Type":"ContainerDied","Data":"26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738"}
Oct 31 01:05:37.154459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.154230    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57878cf6bd-2r9fz" event={"ID":"bf03a56d-1c3f-486b-bb48-df377b3d4768","Type":"ContainerDied","Data":"569d1a90377a1fe92f30145ed98b1e289e9a6acdeed9d58cddeaa1cbad9dc18c"}
Oct 31 01:05:37.154459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.154256    5419 scope.go:122] "RemoveContainer" containerID="26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738"
Oct 31 01:05:37.162083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.162063    5419 scope.go:122] "RemoveContainer" containerID="26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738"
Oct 31 01:05:37.162353 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:37.162337    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738\": container with ID starting with 26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738 not found: ID does not exist" containerID="26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738"
Oct 31 01:05:37.162403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.162362    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738"} err="failed to get container status \"26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738\": rpc error: code = NotFound desc = could not find container \"26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738\": container with ID starting with 26987f7a07f973a92ba6c772a6469a73ad0779c2f7fdefa7c29ece89910d8738 not found: ID does not exist"
Oct 31 01:05:37.316020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.315979    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57878cf6bd-2r9fz"]
Oct 31 01:05:37.320108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:37.320081    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57878cf6bd-2r9fz"]
Oct 31 01:05:38.834388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:05:38.834362    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf03a56d-1c3f-486b-bb48-df377b3d4768" path="/var/lib/kubelet/pods/bf03a56d-1c3f-486b-bb48-df377b3d4768/volumes"
Oct 31 01:05:44.579757 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:05:44.579725    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:06:14.581307 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:06:14.581266    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:06:44.580885 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:06:44.580801    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:07:14.580156 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:07:14.580092    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:07:18.061772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.061742    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"]
Oct 31 01:07:18.062219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.062123    5419 cpu_manager.go:414] "containerMap: removing container" podUID="bf03a56d-1c3f-486b-bb48-df377b3d4768" containerName="console"
Oct 31 01:07:18.062219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.062160    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf03a56d-1c3f-486b-bb48-df377b3d4768" containerName="console"
Oct 31 01:07:18.062327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.062231    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf03a56d-1c3f-486b-bb48-df377b3d4768" containerName="console"
Oct 31 01:07:18.063864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.063848    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.069543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.069518    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.069669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.069558    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ds44\" (UniqueName: \"kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.069714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.069674    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.116594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.116571    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 01:07:18.166035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.166010    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 01:07:18.166447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.166427    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-dwxgt\""
Oct 31 01:07:18.166914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.166895    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"]
Oct 31 01:07:18.167263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.167246    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"]
Oct 31 01:07:18.169313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.169293    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.170261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170240    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrwp5\" (UniqueName: \"kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.170349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170322    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.170421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170350    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ds44\" (UniqueName: \"kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.170421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170387    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.170421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170406    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.170634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170440    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.170695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170682    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.170739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.170704    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.227009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.226986    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ds44\" (UniqueName: \"kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.235284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.235258    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"]
Oct 31 01:07:18.255109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.255087    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5xqz2\""
Oct 31 01:07:18.271257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.271230    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrwp5\" (UniqueName: \"kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.271340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.271283    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.271340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.271310    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.271617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.271597    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.271652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.271632    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.328668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.328592    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrwp5\" (UniqueName: \"kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.374728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.374694    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:18.380900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.380868    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"]
Oct 31 01:07:18.383791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.383771    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.441689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.441656    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-85vx5\""
Oct 31 01:07:18.442420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.442401    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 01:07:18.442500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.442463    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 01:07:18.442500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.442491    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:07:18.442563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.442516    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 01:07:18.442595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.442581    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 01:07:18.442629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.442605    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 01:07:18.446717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.446695    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"]
Oct 31 01:07:18.472798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.472768    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.472798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.472798    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.472974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.472834    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.472974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.472947    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.473037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.472999    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.473115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.473035    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.473115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.473078    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569h5\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.480831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.480815    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:18.573426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573399    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.573592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573432    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.573592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573478    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.573592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573505    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.573592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573538    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.573592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573565    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.573838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.573606    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569h5\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.574349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.574291    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.575857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.575836    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.576244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.576217    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.576473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.576454    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.576519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.576472    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.631040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.630976    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.631233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.631211    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569h5\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5\") pod \"istiod-openshift-gateway-7876bfcb74-d8d99\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.714236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.714205    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:18.748688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.748586    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"]
Oct 31 01:07:18.751251 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:07:18.751214    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a618ee1_b9aa_45fb_8d40_9df4b9c30a43.slice/crio-798c749dd9afe8fc742608c464fa139ffbc92c4e808eee2e4f431229726d786a WatchSource:0}: Error finding container 798c749dd9afe8fc742608c464fa139ffbc92c4e808eee2e4f431229726d786a: Status 404 returned error can't find the container with id 798c749dd9afe8fc742608c464fa139ffbc92c4e808eee2e4f431229726d786a
Oct 31 01:07:18.847939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:18.847911    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"]
Oct 31 01:07:18.850813 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:07:18.850789    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74709cd6_e45e_4727_9136_d84d29d77e0e.slice/crio-d30fa0beb68630b4d17a514d2d05d541cbfea8a510912d1a2d975d1481155864 WatchSource:0}: Error finding container d30fa0beb68630b4d17a514d2d05d541cbfea8a510912d1a2d975d1481155864: Status 404 returned error can't find the container with id d30fa0beb68630b4d17a514d2d05d541cbfea8a510912d1a2d975d1481155864
Oct 31 01:07:19.087233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:19.087201    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"]
Oct 31 01:07:19.090332 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:07:19.090300    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351e3d7a_604c_473a_bdef_feb838d7796b.slice/crio-914bd9e09841ed3fc09fdd0eda86d5327c43af7655880d9e090d2b31c106cb1f WatchSource:0}: Error finding container 914bd9e09841ed3fc09fdd0eda86d5327c43af7655880d9e090d2b31c106cb1f: Status 404 returned error can't find the container with id 914bd9e09841ed3fc09fdd0eda86d5327c43af7655880d9e090d2b31c106cb1f
Oct 31 01:07:19.482151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:19.482112    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" event={"ID":"74709cd6-e45e-4727-9136-d84d29d77e0e","Type":"ContainerStarted","Data":"d30fa0beb68630b4d17a514d2d05d541cbfea8a510912d1a2d975d1481155864"}
Oct 31 01:07:19.483326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:19.483302    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" event={"ID":"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43","Type":"ContainerStarted","Data":"798c749dd9afe8fc742608c464fa139ffbc92c4e808eee2e4f431229726d786a"}
Oct 31 01:07:19.484385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:19.484366    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" event={"ID":"351e3d7a-604c-473a-bdef-feb838d7796b","Type":"ContainerStarted","Data":"914bd9e09841ed3fc09fdd0eda86d5327c43af7655880d9e090d2b31c106cb1f"}
Oct 31 01:07:23.522925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:23.522545    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:07:23.522925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:23.522608    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:07:24.508269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:24.508196    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" event={"ID":"351e3d7a-604c-473a-bdef-feb838d7796b","Type":"ContainerStarted","Data":"5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425"}
Oct 31 01:07:24.508803 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:24.508740    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:24.510515 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:24.510486    5419 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-d8d99 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:07:24.510631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:24.510541    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:07:24.716041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:24.715977    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" podStartSLOduration=2.286477628 podStartE2EDuration="6.715956123s" totalImagesPullingTime="4.429478495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:24.674417096 +0000 UTC m=+165.975373666" watchObservedRunningTime="2026-10-31 01:07:24.715956123 +0000 UTC m=+166.016912701"
Oct 31 01:07:25.513753 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:25.513713    5419 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-d8d99 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:07:25.513921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:25.513776    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:07:26.518045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:26.518014    5419 generic.go:356] "Generic (PLEG): container finished" podID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerID="483a0e2e16c31962b9df4d20e58700aa1c417e2c5085a2c3311da4a3ef61a68b" exitCode=0
Oct 31 01:07:26.518526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:26.518057    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" event={"ID":"74709cd6-e45e-4727-9136-d84d29d77e0e","Type":"ContainerDied","Data":"483a0e2e16c31962b9df4d20e58700aa1c417e2c5085a2c3311da4a3ef61a68b"}
Oct 31 01:07:26.521794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:26.521767    5419 generic.go:356] "Generic (PLEG): container finished" podID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerID="550abfed6ac2876a8f10bacc2d8108eef679275022218f805b56b40457d553f6" exitCode=0
Oct 31 01:07:26.521870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:26.521814    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" event={"ID":"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43","Type":"ContainerDied","Data":"550abfed6ac2876a8f10bacc2d8108eef679275022218f805b56b40457d553f6"}
Oct 31 01:07:26.522761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:26.522738    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:07:28.531028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:28.530992    5419 generic.go:356] "Generic (PLEG): container finished" podID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerID="5ac3793182c65155fecc167a9d133270a0934d0dbb7294acf4a9191c78ceb02a" exitCode=0
Oct 31 01:07:28.531478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:28.531096    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" event={"ID":"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43","Type":"ContainerDied","Data":"5ac3793182c65155fecc167a9d133270a0934d0dbb7294acf4a9191c78ceb02a"}
Oct 31 01:07:30.541195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:30.541131    5419 generic.go:356] "Generic (PLEG): container finished" podID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerID="fc0821d7dde5208f7cc661f72e009aba3746823adc956c6bfca26ffc055260b6" exitCode=0
Oct 31 01:07:30.541721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:30.541236    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" event={"ID":"74709cd6-e45e-4727-9136-d84d29d77e0e","Type":"ContainerDied","Data":"fc0821d7dde5208f7cc661f72e009aba3746823adc956c6bfca26ffc055260b6"}
Oct 31 01:07:36.570015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:36.569977    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" event={"ID":"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43","Type":"ContainerStarted","Data":"654ff559786c4325fb2454913a229a1a2bfc7b900581adc49d042fff1ae43915"}
Oct 31 01:07:36.686039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:36.685922    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" podStartSLOduration=2.460318792 podStartE2EDuration="18.685903538s" totalImagesPullingTime="16.225584746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:36.685524835 +0000 UTC m=+177.986481418" watchObservedRunningTime="2026-10-31 01:07:36.685903538 +0000 UTC m=+177.986860117"
Oct 31 01:07:37.576231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:37.576191    5419 generic.go:356] "Generic (PLEG): container finished" podID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerID="654ff559786c4325fb2454913a229a1a2bfc7b900581adc49d042fff1ae43915" exitCode=0
Oct 31 01:07:37.576687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:37.576277    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" event={"ID":"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43","Type":"ContainerDied","Data":"654ff559786c4325fb2454913a229a1a2bfc7b900581adc49d042fff1ae43915"}
Oct 31 01:07:38.581782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.581742    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" event={"ID":"74709cd6-e45e-4727-9136-d84d29d77e0e","Type":"ContainerStarted","Data":"c7583f71111b22ee3d13a6949ebf90139c18caa754105148153c0842328d60e8"}
Oct 31 01:07:38.698330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.698282    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" podStartSLOduration=2.458920759 podStartE2EDuration="20.698268053s" totalImagesPullingTime="18.239347294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:07:38.696812845 +0000 UTC m=+179.997769436" watchObservedRunningTime="2026-10-31 01:07:38.698268053 +0000 UTC m=+179.999224630"
Oct 31 01:07:38.700577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.700561    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:38.744654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.744628    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle\") pod \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") "
Oct 31 01:07:38.744750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.744662    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44\" (UniqueName: \"kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44\") pod \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") "
Oct 31 01:07:38.744750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.744706    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util\") pod \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\" (UID: \"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43\") "
Oct 31 01:07:38.745223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.745198    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle" pod "1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" (UID: "1a618ee1-b9aa-45fb-8d40-9df4b9c30a43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:38.746854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.746834    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44" pod "1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" (UID: "1a618ee1-b9aa-45fb-8d40-9df4b9c30a43"). InnerVolumeSpecName "kube-api-access-7ds44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:38.750375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.750348    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util" pod "1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" (UID: "1a618ee1-b9aa-45fb-8d40-9df4b9c30a43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:38.845206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.845180    5419 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-bundle\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:38.845332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.845208    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ds44\" (UniqueName: \"kubernetes.io/projected/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-kube-api-access-7ds44\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:38.845332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:38.845221    5419 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a618ee1-b9aa-45fb-8d40-9df4b9c30a43-util\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:39.587243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:39.587203    5419 generic.go:356] "Generic (PLEG): container finished" podID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerID="c7583f71111b22ee3d13a6949ebf90139c18caa754105148153c0842328d60e8" exitCode=0
Oct 31 01:07:39.587686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:39.587293    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" event={"ID":"74709cd6-e45e-4727-9136-d84d29d77e0e","Type":"ContainerDied","Data":"c7583f71111b22ee3d13a6949ebf90139c18caa754105148153c0842328d60e8"}
Oct 31 01:07:39.589153 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:39.589112    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2"
Oct 31 01:07:39.589274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:39.589110    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be718mtn2" event={"ID":"1a618ee1-b9aa-45fb-8d40-9df4b9c30a43","Type":"ContainerDied","Data":"798c749dd9afe8fc742608c464fa139ffbc92c4e808eee2e4f431229726d786a"}
Oct 31 01:07:39.589274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:39.589240    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="798c749dd9afe8fc742608c464fa139ffbc92c4e808eee2e4f431229726d786a"
Oct 31 01:07:40.755063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.755027    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:40.862165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.862063    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle\") pod \"74709cd6-e45e-4727-9136-d84d29d77e0e\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") "
Oct 31 01:07:40.862342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.862169    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5\" (UniqueName: \"kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5\") pod \"74709cd6-e45e-4727-9136-d84d29d77e0e\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") "
Oct 31 01:07:40.862342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.862207    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util\") pod \"74709cd6-e45e-4727-9136-d84d29d77e0e\" (UID: \"74709cd6-e45e-4727-9136-d84d29d77e0e\") "
Oct 31 01:07:40.862882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.862855    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle" pod "74709cd6-e45e-4727-9136-d84d29d77e0e" (UID: "74709cd6-e45e-4727-9136-d84d29d77e0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:40.864394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.864370    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5" pod "74709cd6-e45e-4727-9136-d84d29d77e0e" (UID: "74709cd6-e45e-4727-9136-d84d29d77e0e"). InnerVolumeSpecName "kube-api-access-zrwp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:40.868097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.868072    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util" pod "74709cd6-e45e-4727-9136-d84d29d77e0e" (UID: "74709cd6-e45e-4727-9136-d84d29d77e0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:07:40.962873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.962836    5419 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-bundle\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:40.962873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.962864    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrwp5\" (UniqueName: \"kubernetes.io/projected/74709cd6-e45e-4727-9136-d84d29d77e0e-kube-api-access-zrwp5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:40.962873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:40.962873    5419 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74709cd6-e45e-4727-9136-d84d29d77e0e-util\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:41.597606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:41.597565    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps" event={"ID":"74709cd6-e45e-4727-9136-d84d29d77e0e","Type":"ContainerDied","Data":"d30fa0beb68630b4d17a514d2d05d541cbfea8a510912d1a2d975d1481155864"}
Oct 31 01:07:41.597784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:41.597640    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30fa0beb68630b4d17a514d2d05d541cbfea8a510912d1a2d975d1481155864"
Oct 31 01:07:41.597784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:41.597581    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdqcps"
Oct 31 01:07:44.580203 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:07:44.580165    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:07:59.876026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.875992    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"]
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876395    5419 cpu_manager.go:414] "containerMap: removing container" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="extract"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876413    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="extract"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876428    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="util"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876437    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="util"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876451    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="pull"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876458    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="pull"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876477    5419 cpu_manager.go:414] "containerMap: removing container" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="util"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876483    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="util"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876492    5419 cpu_manager.go:414] "containerMap: removing container" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="pull"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876498    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="pull"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876514    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="extract"
Oct 31 01:07:59.876572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876520    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="extract"
Oct 31 01:07:59.877172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876602    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a618ee1-b9aa-45fb-8d40-9df4b9c30a43" containerName="extract"
Oct 31 01:07:59.877172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.876617    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="74709cd6-e45e-4727-9136-d84d29d77e0e" containerName="extract"
Oct 31 01:07:59.878493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.878474    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:07:59.954040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.954007    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 01:07:59.954224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.954072    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 01:07:59.989487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.989389    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 01:07:59.989487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.989400    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 01:07:59.989487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.989487    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9qxzz\""
Oct 31 01:07:59.989774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:07:59.989436    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 01:08:00.004562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.004526    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"]
Oct 31 01:08:00.018338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.018313    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.018454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.018346    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.018454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.018372    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwcgp\" (UniqueName: \"kubernetes.io/projected/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-kube-api-access-mwcgp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.018454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.018443    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.018550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.018507    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.018585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.018562    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.119442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.119398    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.119442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.119456    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.119673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.119584    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.119673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.119613    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.119673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.119634    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwcgp\" (UniqueName: \"kubernetes.io/projected/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-kube-api-access-mwcgp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.119673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.119651    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.120070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.120040    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-manifests\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.122068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.122038    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-tmp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.122280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.122260    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.122346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.122285    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-metrics-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.122346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.122324    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.180186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.180104    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwcgp\" (UniqueName: \"kubernetes.io/projected/7ba21b02-c733-4fd0-b735-8d6cbc2927f4-kube-api-access-mwcgp\") pod \"opendatahub-operator-controller-manager-559fd7ff56-hbjp8\" (UID: \"7ba21b02-c733-4fd0-b735-8d6cbc2927f4\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.189952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.189928    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:00.726908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:00.726878    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"]
Oct 31 01:08:00.729925 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:08:00.729900    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba21b02_c733_4fd0_b735_8d6cbc2927f4.slice/crio-fdecf82d572da2e31ab9a582a997abf1c675cb58e06357a569418bb04b8a4977 WatchSource:0}: Error finding container fdecf82d572da2e31ab9a582a997abf1c675cb58e06357a569418bb04b8a4977: Status 404 returned error can't find the container with id fdecf82d572da2e31ab9a582a997abf1c675cb58e06357a569418bb04b8a4977
Oct 31 01:08:01.667153 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:01.667106    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8" event={"ID":"7ba21b02-c733-4fd0-b735-8d6cbc2927f4","Type":"ContainerStarted","Data":"fdecf82d572da2e31ab9a582a997abf1c675cb58e06357a569418bb04b8a4977"}
Oct 31 01:08:04.681386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:04.681352    5419 generic.go:356] "Generic (PLEG): container finished" podID="7ba21b02-c733-4fd0-b735-8d6cbc2927f4" containerID="41d2af4f46596e27e522d771546ffddde17a0ddfc1a6e6931076d7b8ff811c25" exitCode=0
Oct 31 01:08:04.681787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:04.681401    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8" event={"ID":"7ba21b02-c733-4fd0-b735-8d6cbc2927f4","Type":"ContainerDied","Data":"41d2af4f46596e27e522d771546ffddde17a0ddfc1a6e6931076d7b8ff811c25"}
Oct 31 01:08:05.687735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:05.687697    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8" event={"ID":"7ba21b02-c733-4fd0-b735-8d6cbc2927f4","Type":"ContainerStarted","Data":"175f3c4750354199605c23ce938a443a5fa3dbdf9eb131b27e643f1e4ee2ab06"}
Oct 31 01:08:05.688160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:05.687775    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:05.925117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:05.925014    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8" podStartSLOduration=2.31389528 podStartE2EDuration="6.925000103s" totalImagesPullingTime="4.124797735s" totalInitContainerRuntime="486.307088ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:07:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:05.875643756 +0000 UTC m=+207.176600333" watchObservedRunningTime="2026-10-31 01:08:05.925000103 +0000 UTC m=+207.225956680"
Oct 31 01:08:14.580202 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:08:14.580157    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:08:16.693186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:16.693131    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-559fd7ff56-hbjp8"
Oct 31 01:08:24.586953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.586918    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"]
Oct 31 01:08:24.590261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.590236    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:24.610389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.610362    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmvq\" (UniqueName: \"kubernetes.io/projected/3f1e8c92-0009-4922-824d-162960b28060-kube-api-access-ldmvq\") pod \"authorino-operator-7b7bb4bc77-v8knf\" (UID: \"3f1e8c92-0009-4922-824d-162960b28060\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:24.647595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.647558    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 01:08:24.697123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.697083    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 01:08:24.697123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.697083    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xkxtg\""
Oct 31 01:08:24.697956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.697930    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"]
Oct 31 01:08:24.711629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.711604    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmvq\" (UniqueName: \"kubernetes.io/projected/3f1e8c92-0009-4922-824d-162960b28060-kube-api-access-ldmvq\") pod \"authorino-operator-7b7bb4bc77-v8knf\" (UID: \"3f1e8c92-0009-4922-824d-162960b28060\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:24.769831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.769793    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmvq\" (UniqueName: \"kubernetes.io/projected/3f1e8c92-0009-4922-824d-162960b28060-kube-api-access-ldmvq\") pod \"authorino-operator-7b7bb4bc77-v8knf\" (UID: \"3f1e8c92-0009-4922-824d-162960b28060\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:24.901843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:24.901757    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:25.273505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:25.273474    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"]
Oct 31 01:08:25.276500 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:08:25.276473    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1e8c92_0009_4922_824d_162960b28060.slice/crio-ee9ed70aff614e93374f4b5b14dd4deb795dadd0261c800ac81781976842cf22 WatchSource:0}: Error finding container ee9ed70aff614e93374f4b5b14dd4deb795dadd0261c800ac81781976842cf22: Status 404 returned error can't find the container with id ee9ed70aff614e93374f4b5b14dd4deb795dadd0261c800ac81781976842cf22
Oct 31 01:08:25.766536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:25.766505    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf" event={"ID":"3f1e8c92-0009-4922-824d-162960b28060","Type":"ContainerStarted","Data":"ee9ed70aff614e93374f4b5b14dd4deb795dadd0261c800ac81781976842cf22"}
Oct 31 01:08:28.782659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:28.782631    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf" event={"ID":"3f1e8c92-0009-4922-824d-162960b28060","Type":"ContainerStarted","Data":"861cc735f70fb1933523f171e8c527f79b5fa5243d1179ed2534da5d9bace0fa"}
Oct 31 01:08:28.783062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:28.782761    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:28.933522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:28.933466    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf" podStartSLOduration=1.500098054 podStartE2EDuration="4.933451485s" totalImagesPullingTime="3.433353431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:28.932193515 +0000 UTC m=+230.233150096" watchObservedRunningTime="2026-10-31 01:08:28.933451485 +0000 UTC m=+230.234408106"
Oct 31 01:08:39.789105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:39.789068    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-v8knf"
Oct 31 01:08:44.579951 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:08:44.579915    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:08:51.845120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.845088    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"]
Oct 31 01:08:51.848529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.848512    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:51.900348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.900312    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 01:08:51.923960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.923927    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3410a6d1-0d32-4001-8588-fca0dea3fbaf-tmp\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:51.924094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.923986    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3410a6d1-0d32-4001-8588-fca0dea3fbaf-tls-certs\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:51.924094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.924075    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpp5\" (UniqueName: \"kubernetes.io/projected/3410a6d1-0d32-4001-8588-fca0dea3fbaf-kube-api-access-kvpp5\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:51.949469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.949437    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 01:08:51.949605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.949524    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xnxmt\""
Oct 31 01:08:51.950389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:51.950368    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"]
Oct 31 01:08:52.025527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.025491    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3410a6d1-0d32-4001-8588-fca0dea3fbaf-tls-certs\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.025712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.025549    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpp5\" (UniqueName: \"kubernetes.io/projected/3410a6d1-0d32-4001-8588-fca0dea3fbaf-kube-api-access-kvpp5\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.025712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.025626    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3410a6d1-0d32-4001-8588-fca0dea3fbaf-tmp\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.028234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.028207    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3410a6d1-0d32-4001-8588-fca0dea3fbaf-tls-certs\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.028234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.028221    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3410a6d1-0d32-4001-8588-fca0dea3fbaf-tmp\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.084643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.084607    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpp5\" (UniqueName: \"kubernetes.io/projected/3410a6d1-0d32-4001-8588-fca0dea3fbaf-kube-api-access-kvpp5\") pod \"kube-auth-proxy-7ff59956fc-sgs8x\" (UID: \"3410a6d1-0d32-4001-8588-fca0dea3fbaf\") " pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.159704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.159625    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"
Oct 31 01:08:52.529538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.529504    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x"]
Oct 31 01:08:52.532878 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:08:52.532850    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3410a6d1_0d32_4001_8588_fca0dea3fbaf.slice/crio-935107a3662248ed8472a121e327cadc890b312d8d796da0840a7f5e7d80a0a2 WatchSource:0}: Error finding container 935107a3662248ed8472a121e327cadc890b312d8d796da0840a7f5e7d80a0a2: Status 404 returned error can't find the container with id 935107a3662248ed8472a121e327cadc890b312d8d796da0840a7f5e7d80a0a2
Oct 31 01:08:52.880228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:52.880129    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x" event={"ID":"3410a6d1-0d32-4001-8588-fca0dea3fbaf","Type":"ContainerStarted","Data":"935107a3662248ed8472a121e327cadc890b312d8d796da0840a7f5e7d80a0a2"}
Oct 31 01:08:55.895015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:55.894981    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x" event={"ID":"3410a6d1-0d32-4001-8588-fca0dea3fbaf","Type":"ContainerStarted","Data":"0d3a4d03f0aa7ee117bb9f49e9cefb727b540044dc028ee574aa9eb6c4d8c95f"}
Oct 31 01:08:56.019683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:08:56.019638    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ff59956fc-sgs8x" podStartSLOduration=2.16718414 podStartE2EDuration="5.019625843s" totalImagesPullingTime="2.852441703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:56.019227005 +0000 UTC m=+257.320183582" watchObservedRunningTime="2026-10-31 01:08:56.019625843 +0000 UTC m=+257.320582420"
Oct 31 01:09:09.409027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.408990    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lf4j2"]
Oct 31 01:09:09.413088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.413069    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:09.470604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.470576    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lf4j2"]
Oct 31 01:09:09.515132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.515106    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vl92q\""
Oct 31 01:09:09.572627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.572588    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcvp\" (UniqueName: \"kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp\") pod \"authorino-57f7f8659f-lf4j2\" (UID: \"eb637376-27eb-49ff-85f4-fe31b5c1e36c\") " pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:09.673677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.673605    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcvp\" (UniqueName: \"kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp\") pod \"authorino-57f7f8659f-lf4j2\" (UID: \"eb637376-27eb-49ff-85f4-fe31b5c1e36c\") " pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:09.729770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.729735    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcvp\" (UniqueName: \"kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp\") pod \"authorino-57f7f8659f-lf4j2\" (UID: \"eb637376-27eb-49ff-85f4-fe31b5c1e36c\") " pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:09.743570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:09.743527    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:10.107745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:10.107700    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lf4j2"]
Oct 31 01:09:10.111992 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:09:10.111964    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb637376_27eb_49ff_85f4_fe31b5c1e36c.slice/crio-4e960fb896b9e11ff1f7a86590b573c44359190e961c6823f375d6354e566950 WatchSource:0}: Error finding container 4e960fb896b9e11ff1f7a86590b573c44359190e961c6823f375d6354e566950: Status 404 returned error can't find the container with id 4e960fb896b9e11ff1f7a86590b573c44359190e961c6823f375d6354e566950
Oct 31 01:09:10.953552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:10.953514    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lf4j2" event={"ID":"eb637376-27eb-49ff-85f4-fe31b5c1e36c","Type":"ContainerStarted","Data":"4e960fb896b9e11ff1f7a86590b573c44359190e961c6823f375d6354e566950"}
Oct 31 01:09:12.459498 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.459461    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m94b9"]
Oct 31 01:09:12.461865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.461841    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:09:12.464044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.464013    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:09:12.466166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.466111    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.518397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.518368    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m94b9"]
Oct 31 01:09:12.562598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.562575    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 01:09:12.562598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.562584    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6qzfw\""
Oct 31 01:09:12.563777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.563752    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:09:12.599352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.599324    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93d3c753-4f46-49ca-9f45-bfdfd1679571-config-file\") pod \"limitador-limitador-7c9b87d574-vlvpt\" (UID: \"93d3c753-4f46-49ca-9f45-bfdfd1679571\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.599462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.599394    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhsjf\" (UniqueName: \"kubernetes.io/projected/93d3c753-4f46-49ca-9f45-bfdfd1679571-kube-api-access-qhsjf\") pod \"limitador-limitador-7c9b87d574-vlvpt\" (UID: \"93d3c753-4f46-49ca-9f45-bfdfd1679571\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.599509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.599491    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjdh\" (UniqueName: \"kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh\") pod \"authorino-56c6c9ddd5-m94b9\" (UID: \"2556438b-6208-4049-b72a-da6a62a82b14\") " pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:09:12.702727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.701023    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhsjf\" (UniqueName: \"kubernetes.io/projected/93d3c753-4f46-49ca-9f45-bfdfd1679571-kube-api-access-qhsjf\") pod \"limitador-limitador-7c9b87d574-vlvpt\" (UID: \"93d3c753-4f46-49ca-9f45-bfdfd1679571\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.702727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.701173    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjdh\" (UniqueName: \"kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh\") pod \"authorino-56c6c9ddd5-m94b9\" (UID: \"2556438b-6208-4049-b72a-da6a62a82b14\") " pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:09:12.702727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.701244    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93d3c753-4f46-49ca-9f45-bfdfd1679571-config-file\") pod \"limitador-limitador-7c9b87d574-vlvpt\" (UID: \"93d3c753-4f46-49ca-9f45-bfdfd1679571\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.702727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.702250    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/93d3c753-4f46-49ca-9f45-bfdfd1679571-config-file\") pod \"limitador-limitador-7c9b87d574-vlvpt\" (UID: \"93d3c753-4f46-49ca-9f45-bfdfd1679571\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.757718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.757683    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhsjf\" (UniqueName: \"kubernetes.io/projected/93d3c753-4f46-49ca-9f45-bfdfd1679571-kube-api-access-qhsjf\") pod \"limitador-limitador-7c9b87d574-vlvpt\" (UID: \"93d3c753-4f46-49ca-9f45-bfdfd1679571\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.757877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.757784    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjdh\" (UniqueName: \"kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh\") pod \"authorino-56c6c9ddd5-m94b9\" (UID: \"2556438b-6208-4049-b72a-da6a62a82b14\") " pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:09:12.763437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.763413    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:09:12.779473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.779456    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:09:12.788345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.788321    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:12.963624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:12.963579    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lf4j2" event={"ID":"eb637376-27eb-49ff-85f4-fe31b5c1e36c","Type":"ContainerStarted","Data":"79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691"}
Oct 31 01:09:13.075953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:13.075851    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-lf4j2" podStartSLOduration=1.631736498 podStartE2EDuration="4.075836606s" totalImagesPullingTime="2.444100108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:13.075246597 +0000 UTC m=+274.376203189" watchObservedRunningTime="2026-10-31 01:09:13.075836606 +0000 UTC m=+274.376793240"
Oct 31 01:09:13.203400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:13.203357    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m94b9"]
Oct 31 01:09:13.206318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:13.206295    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:09:13.208282 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:09:13.208256    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2556438b_6208_4049_b72a_da6a62a82b14.slice/crio-b31455d28487c127c29a4138fbff96a5e575e2f18b2c614540fd1ad57e8c77a5 WatchSource:0}: Error finding container b31455d28487c127c29a4138fbff96a5e575e2f18b2c614540fd1ad57e8c77a5: Status 404 returned error can't find the container with id b31455d28487c127c29a4138fbff96a5e575e2f18b2c614540fd1ad57e8c77a5
Oct 31 01:09:13.209065 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:09:13.209044    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d3c753_4f46_49ca_9f45_bfdfd1679571.slice/crio-bcbd1a583da2262478e11f5ec17bd62a5d69bf95344ae0c6b3068c21032aef23 WatchSource:0}: Error finding container bcbd1a583da2262478e11f5ec17bd62a5d69bf95344ae0c6b3068c21032aef23: Status 404 returned error can't find the container with id bcbd1a583da2262478e11f5ec17bd62a5d69bf95344ae0c6b3068c21032aef23
Oct 31 01:09:13.969702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:13.969663    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9" event={"ID":"2556438b-6208-4049-b72a-da6a62a82b14","Type":"ContainerStarted","Data":"b31455d28487c127c29a4138fbff96a5e575e2f18b2c614540fd1ad57e8c77a5"}
Oct 31 01:09:13.970859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:13.970832    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt" event={"ID":"93d3c753-4f46-49ca-9f45-bfdfd1679571","Type":"ContainerStarted","Data":"bcbd1a583da2262478e11f5ec17bd62a5d69bf95344ae0c6b3068c21032aef23"}
Oct 31 01:09:14.597672 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:09:14.597641    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:09:14.976668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:14.976625    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9" event={"ID":"2556438b-6208-4049-b72a-da6a62a82b14","Type":"ContainerStarted","Data":"3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867"}
Oct 31 01:09:15.089668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.089606    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9" podStartSLOduration=1.759955068 podStartE2EDuration="3.089590376s" totalImagesPullingTime="1.329635308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:15.088809418 +0000 UTC m=+276.389766001" watchObservedRunningTime="2026-10-31 01:09:15.089590376 +0000 UTC m=+276.390546984"
Oct 31 01:09:15.139112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.139082    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lf4j2"]
Oct 31 01:09:15.139419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.139391    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-lf4j2" podUID="eb637376-27eb-49ff-85f4-fe31b5c1e36c" containerName="authorino" containerID="cri-o://79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691" gracePeriod=30
Oct 31 01:09:15.378654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.378628    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:15.525954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.525879    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp\" (UniqueName: \"kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp\") pod \"eb637376-27eb-49ff-85f4-fe31b5c1e36c\" (UID: \"eb637376-27eb-49ff-85f4-fe31b5c1e36c\") "
Oct 31 01:09:15.528186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.528150    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp" pod "eb637376-27eb-49ff-85f4-fe31b5c1e36c" (UID: "eb637376-27eb-49ff-85f4-fe31b5c1e36c"). InnerVolumeSpecName "kube-api-access-cvcvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:15.626733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.626705    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvcvp\" (UniqueName: \"kubernetes.io/projected/eb637376-27eb-49ff-85f4-fe31b5c1e36c-kube-api-access-cvcvp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:15.981666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.981635    5419 generic.go:356] "Generic (PLEG): container finished" podID="eb637376-27eb-49ff-85f4-fe31b5c1e36c" containerID="79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691" exitCode=0
Oct 31 01:09:15.982115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.981691    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-lf4j2"
Oct 31 01:09:15.982115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.981705    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lf4j2" event={"ID":"eb637376-27eb-49ff-85f4-fe31b5c1e36c","Type":"ContainerDied","Data":"79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691"}
Oct 31 01:09:15.982115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.981747    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-lf4j2" event={"ID":"eb637376-27eb-49ff-85f4-fe31b5c1e36c","Type":"ContainerDied","Data":"4e960fb896b9e11ff1f7a86590b573c44359190e961c6823f375d6354e566950"}
Oct 31 01:09:15.982115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.981772    5419 scope.go:122] "RemoveContainer" containerID="79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691"
Oct 31 01:09:15.983424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.983397    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt" event={"ID":"93d3c753-4f46-49ca-9f45-bfdfd1679571","Type":"ContainerStarted","Data":"4dfa6f94feacc0031c312680b90180ef9ffc054173c3da8216e8a540cd665e6e"}
Oct 31 01:09:15.990123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.990105    5419 scope.go:122] "RemoveContainer" containerID="79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691"
Oct 31 01:09:15.990456 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:09:15.990438    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691\": container with ID starting with 79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691 not found: ID does not exist" containerID="79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691"
Oct 31 01:09:15.990504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:15.990465    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691"} err="failed to get container status \"79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691\": rpc error: code = NotFound desc = could not find container \"79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691\": container with ID starting with 79af96fa27d8ce6210d116e21b2a7314bcde271ad495d775651395ee3075b691 not found: ID does not exist"
Oct 31 01:09:16.235651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:16.235526    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt" podStartSLOduration=2.350237598 podStartE2EDuration="4.235508587s" totalImagesPullingTime="1.885270989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:09:16.235023524 +0000 UTC m=+277.535980101" watchObservedRunningTime="2026-10-31 01:09:16.235508587 +0000 UTC m=+277.536465164"
Oct 31 01:09:16.343459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:16.343429    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lf4j2"]
Oct 31 01:09:16.348691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:16.348668    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-lf4j2"]
Oct 31 01:09:16.835352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:16.835318    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb637376-27eb-49ff-85f4-fe31b5c1e36c" path="/var/lib/kubelet/pods/eb637376-27eb-49ff-85f4-fe31b5c1e36c/volumes"
Oct 31 01:09:16.989772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:16.989733    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:27.994716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:09:27.994632    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"
Oct 31 01:09:44.581900 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:09:44.581855    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:10:03.990382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:03.990345    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"]
Oct 31 01:10:03.990813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:03.990797    5419 cpu_manager.go:414] "containerMap: removing container" podUID="eb637376-27eb-49ff-85f4-fe31b5c1e36c" containerName="authorino"
Oct 31 01:10:03.990853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:03.990814    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb637376-27eb-49ff-85f4-fe31b5c1e36c" containerName="authorino"
Oct 31 01:10:03.990913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:03.990899    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb637376-27eb-49ff-85f4-fe31b5c1e36c" containerName="authorino"
Oct 31 01:10:03.993074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:03.993059    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.097349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.097314    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"]
Oct 31 01:10:04.112194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112171    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.112315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112198    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.112315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112221    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca41a23-4748-4e1d-9527-e635da20d712-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.112315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112247    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca41a23-4748-4e1d-9527-e635da20d712-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.112446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112333    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eca41a23-4748-4e1d-9527-e635da20d712-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.112446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112417    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5rb\" (UniqueName: \"kubernetes.io/projected/eca41a23-4748-4e1d-9527-e635da20d712-kube-api-access-kc5rb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.112508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.112455    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213638    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213672    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213695    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca41a23-4748-4e1d-9527-e635da20d712-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213722    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca41a23-4748-4e1d-9527-e635da20d712-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213742    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eca41a23-4748-4e1d-9527-e635da20d712-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213866    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5rb\" (UniqueName: \"kubernetes.io/projected/eca41a23-4748-4e1d-9527-e635da20d712-kube-api-access-kc5rb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.213979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.213946    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.214809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.214781    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eca41a23-4748-4e1d-9527-e635da20d712-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.216235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.216216    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eca41a23-4748-4e1d-9527-e635da20d712-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.216374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.216356    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.216655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.216635    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.216761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.216693    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eca41a23-4748-4e1d-9527-e635da20d712-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.271447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.271377    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eca41a23-4748-4e1d-9527-e635da20d712-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.271591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.271543    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5rb\" (UniqueName: \"kubernetes.io/projected/eca41a23-4748-4e1d-9527-e635da20d712-kube-api-access-kc5rb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l6v7x\" (UID: \"eca41a23-4748-4e1d-9527-e635da20d712\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.303562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.303512    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:04.736979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.736949    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"]
Oct 31 01:10:04.739967 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:10:04.739936    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca41a23_4748_4e1d_9527_e635da20d712.slice/crio-36f268fe5d893ee0673690b8e9f26a0993894b1c0cd5e88a220e5588364e701f WatchSource:0}: Error finding container 36f268fe5d893ee0673690b8e9f26a0993894b1c0cd5e88a220e5588364e701f: Status 404 returned error can't find the container with id 36f268fe5d893ee0673690b8e9f26a0993894b1c0cd5e88a220e5588364e701f
Oct 31 01:10:04.741903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.741873    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:10:04.741978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:04.741939    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:10:05.179310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:05.179212    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x" event={"ID":"eca41a23-4748-4e1d-9527-e635da20d712","Type":"ContainerStarted","Data":"197d32886e69e928291b3568d13a92afdf61be7aa5417c0b1940eb324390385b"}
Oct 31 01:10:05.179310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:05.179280    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:05.179310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:05.179300    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x" event={"ID":"eca41a23-4748-4e1d-9527-e635da20d712","Type":"ContainerStarted","Data":"36f268fe5d893ee0673690b8e9f26a0993894b1c0cd5e88a220e5588364e701f"}
Oct 31 01:10:05.180959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:05.180928    5419 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l6v7x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:10:05.181092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:05.180976    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x" podUID="eca41a23-4748-4e1d-9527-e635da20d712" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:10:05.396805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:05.396742    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x" podStartSLOduration=2.396727833 podStartE2EDuration="2.396727833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:05.359223954 +0000 UTC m=+326.660180531" watchObservedRunningTime="2026-10-31 01:10:05.396727833 +0000 UTC m=+326.697684410"
Oct 31 01:10:06.184228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:06.184185    5419 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l6v7x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:10:06.184696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:06.184253    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x" podUID="eca41a23-4748-4e1d-9527-e635da20d712" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:10:09.184889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.184861    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l6v7x"
Oct 31 01:10:09.391310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.391277    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"]
Oct 31 01:10:09.391782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.391727    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" containerName="discovery" containerID="cri-o://5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425" gracePeriod=30
Oct 31 01:10:09.642067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.642040    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:10:09.766777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.766747    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.766958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.766802    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.766958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.766909    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.766958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.766934    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.767117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.766966    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.767117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.766995    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.767117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.767023    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5\") pod \"351e3d7a-604c-473a-bdef-feb838d7796b\" (UID: \"351e3d7a-604c-473a-bdef-feb838d7796b\") "
Oct 31 01:10:09.767510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.767479    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:09.770456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.770425    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:09.770570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.770519    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "kube-api-access-569h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:09.770570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.770514    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:09.770682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.770568    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:09.770682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.770639    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:09.771433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.771414    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig" pod "351e3d7a-604c-473a-bdef-feb838d7796b" (UID: "351e3d7a-604c-473a-bdef-feb838d7796b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:09.868423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868390    5419 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-ca-configmap\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:09.868423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868417    5419 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-csr-dns-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:09.868423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868428    5419 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-cacerts\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:09.868648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868436    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:09.868648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868445    5419 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/351e3d7a-604c-473a-bdef-feb838d7796b-local-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:09.868648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868453    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569h5\" (UniqueName: \"kubernetes.io/projected/351e3d7a-604c-473a-bdef-feb838d7796b-kube-api-access-569h5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:09.868648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:09.868462    5419 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/351e3d7a-604c-473a-bdef-feb838d7796b-istio-kubeconfig\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:10.200637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.200550    5419 generic.go:356] "Generic (PLEG): container finished" podID="351e3d7a-604c-473a-bdef-feb838d7796b" containerID="5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425" exitCode=0
Oct 31 01:10:10.200637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.200604    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"
Oct 31 01:10:10.201081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.200628    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" event={"ID":"351e3d7a-604c-473a-bdef-feb838d7796b","Type":"ContainerDied","Data":"5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425"}
Oct 31 01:10:10.201081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.200686    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99" event={"ID":"351e3d7a-604c-473a-bdef-feb838d7796b","Type":"ContainerDied","Data":"914bd9e09841ed3fc09fdd0eda86d5327c43af7655880d9e090d2b31c106cb1f"}
Oct 31 01:10:10.201081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.200707    5419 scope.go:122] "RemoveContainer" containerID="5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425"
Oct 31 01:10:10.209546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.209526    5419 scope.go:122] "RemoveContainer" containerID="5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425"
Oct 31 01:10:10.209797 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:10:10.209780    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425\": container with ID starting with 5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425 not found: ID does not exist" containerID="5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425"
Oct 31 01:10:10.209851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.209806    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425"} err="failed to get container status \"5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425\": rpc error: code = NotFound desc = could not find container \"5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425\": container with ID starting with 5090a9f98dc902028f3a38ee0089bb0772fd09b3015b17c12c3186c492861425 not found: ID does not exist"
Oct 31 01:10:10.361938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.361903    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"]
Oct 31 01:10:10.411451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.411415    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-d8d99"]
Oct 31 01:10:10.835459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:10.835424    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" path="/var/lib/kubelet/pods/351e3d7a-604c-473a-bdef-feb838d7796b/volumes"
Oct 31 01:10:14.580200 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:10:14.580167    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:10:44.580501 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:10:44.580458    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:10:54.280077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:10:54.279985    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:11:03.735471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.735438    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4pj7h"]
Oct 31 01:11:03.735863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.735741    5419 cpu_manager.go:414] "containerMap: removing container" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" containerName="discovery"
Oct 31 01:11:03.735863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.735752    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" containerName="discovery"
Oct 31 01:11:03.735863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.735821    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="351e3d7a-604c-473a-bdef-feb838d7796b" containerName="discovery"
Oct 31 01:11:03.737456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.737440    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:03.770983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.770948    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfrl\" (UniqueName: \"kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl\") pod \"authorino-6f6944bcdd-4pj7h\" (UID: \"40d1f3fb-ec45-43dd-b933-6367f1bc93a9\") " pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:03.794381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.794351    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4pj7h"]
Oct 31 01:11:03.872292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.872255    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfrl\" (UniqueName: \"kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl\") pod \"authorino-6f6944bcdd-4pj7h\" (UID: \"40d1f3fb-ec45-43dd-b933-6367f1bc93a9\") " pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:03.933680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:03.933642    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfrl\" (UniqueName: \"kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl\") pod \"authorino-6f6944bcdd-4pj7h\" (UID: \"40d1f3fb-ec45-43dd-b933-6367f1bc93a9\") " pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:04.047448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.047380    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:04.097638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.097610    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4pj7h"]
Oct 31 01:11:04.121848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.121817    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d5979fb47-wtnx9"]
Oct 31 01:11:04.126008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.125983    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:04.174571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.174533    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sck9\" (UniqueName: \"kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9\") pod \"authorino-6d5979fb47-wtnx9\" (UID: \"c88b8389-01e2-47a6-bda6-d50b957cd4cf\") " pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:04.186073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.186040    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d5979fb47-wtnx9"]
Oct 31 01:11:04.275969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.275931    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sck9\" (UniqueName: \"kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9\") pod \"authorino-6d5979fb47-wtnx9\" (UID: \"c88b8389-01e2-47a6-bda6-d50b957cd4cf\") " pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:04.332364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.332290    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sck9\" (UniqueName: \"kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9\") pod \"authorino-6d5979fb47-wtnx9\" (UID: \"c88b8389-01e2-47a6-bda6-d50b957cd4cf\") " pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:04.413955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.413921    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4pj7h"]
Oct 31 01:11:04.417158 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:11:04.417109    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d1f3fb_ec45_43dd_b933_6367f1bc93a9.slice/crio-31ad891b770de793dc949306fae449f85f4271497beface1201fc3a617ee312a WatchSource:0}: Error finding container 31ad891b770de793dc949306fae449f85f4271497beface1201fc3a617ee312a: Status 404 returned error can't find the container with id 31ad891b770de793dc949306fae449f85f4271497beface1201fc3a617ee312a
Oct 31 01:11:04.418568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.418546    5419 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:11:04.438788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.438766    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:04.549465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.549436    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d5979fb47-wtnx9"]
Oct 31 01:11:04.575155 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.575104    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57c84b77c7-h6nf2"]
Oct 31 01:11:04.577447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.577422    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.629351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.629325    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 01:11:04.633178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.633155    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c84b77c7-h6nf2"]
Oct 31 01:11:04.678681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.678653    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkd5\" (UniqueName: \"kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5\") pod \"authorino-57c84b77c7-h6nf2\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") " pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.678809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.678708    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert\") pod \"authorino-57c84b77c7-h6nf2\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") " pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.779847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.779815    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkd5\" (UniqueName: \"kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5\") pod \"authorino-57c84b77c7-h6nf2\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") " pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.780357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.779881    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert\") pod \"authorino-57c84b77c7-h6nf2\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") " pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.782489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.782465    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert\") pod \"authorino-57c84b77c7-h6nf2\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") " pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.802058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.802030    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d5979fb47-wtnx9"]
Oct 31 01:11:04.805210 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:11:04.805174    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc88b8389_01e2_47a6_bda6_d50b957cd4cf.slice/crio-0e9dce3845d95176bcd2f92b8f14f4a5f0ceb6aa8928d6e0b44bf1f5b5111cc1 WatchSource:0}: Error finding container 0e9dce3845d95176bcd2f92b8f14f4a5f0ceb6aa8928d6e0b44bf1f5b5111cc1: Status 404 returned error can't find the container with id 0e9dce3845d95176bcd2f92b8f14f4a5f0ceb6aa8928d6e0b44bf1f5b5111cc1
Oct 31 01:11:04.836669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.836604    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkd5\" (UniqueName: \"kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5\") pod \"authorino-57c84b77c7-h6nf2\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") " pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:04.887509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:04.887480    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:11:05.254992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:05.254963    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c84b77c7-h6nf2"]
Oct 31 01:11:05.258085 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:11:05.258059    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf8c0fcb_20b9_4016_aebd_637adc253c6c.slice/crio-4085ef869deebff3b23c19009f101d5e5b46b4395d6634a8af3675cbc44b8627 WatchSource:0}: Error finding container 4085ef869deebff3b23c19009f101d5e5b46b4395d6634a8af3675cbc44b8627: Status 404 returned error can't find the container with id 4085ef869deebff3b23c19009f101d5e5b46b4395d6634a8af3675cbc44b8627
Oct 31 01:11:05.420971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:05.420937    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5979fb47-wtnx9" event={"ID":"c88b8389-01e2-47a6-bda6-d50b957cd4cf","Type":"ContainerStarted","Data":"0e9dce3845d95176bcd2f92b8f14f4a5f0ceb6aa8928d6e0b44bf1f5b5111cc1"}
Oct 31 01:11:05.422104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:05.422077    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c84b77c7-h6nf2" event={"ID":"cf8c0fcb-20b9-4016-aebd-637adc253c6c","Type":"ContainerStarted","Data":"4085ef869deebff3b23c19009f101d5e5b46b4395d6634a8af3675cbc44b8627"}
Oct 31 01:11:05.423180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:05.423158    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h" event={"ID":"40d1f3fb-ec45-43dd-b933-6367f1bc93a9","Type":"ContainerStarted","Data":"31ad891b770de793dc949306fae449f85f4271497beface1201fc3a617ee312a"}
Oct 31 01:11:06.428694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:06.428661    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h" event={"ID":"40d1f3fb-ec45-43dd-b933-6367f1bc93a9","Type":"ContainerStarted","Data":"5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0"}
Oct 31 01:11:06.429110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:06.428728    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h" podUID="40d1f3fb-ec45-43dd-b933-6367f1bc93a9" containerName="authorino" containerID="cri-o://5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0" gracePeriod=30
Oct 31 01:11:06.679461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:06.679431    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:06.797257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:06.797228    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl\" (UniqueName: \"kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl\") pod \"40d1f3fb-ec45-43dd-b933-6367f1bc93a9\" (UID: \"40d1f3fb-ec45-43dd-b933-6367f1bc93a9\") "
Oct 31 01:11:06.799629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:06.799600    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl" pod "40d1f3fb-ec45-43dd-b933-6367f1bc93a9" (UID: "40d1f3fb-ec45-43dd-b933-6367f1bc93a9"). InnerVolumeSpecName "kube-api-access-lxfrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:06.897956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:06.897900    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxfrl\" (UniqueName: \"kubernetes.io/projected/40d1f3fb-ec45-43dd-b933-6367f1bc93a9-kube-api-access-lxfrl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.435081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.435038    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c84b77c7-h6nf2" event={"ID":"cf8c0fcb-20b9-4016-aebd-637adc253c6c","Type":"ContainerStarted","Data":"10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c"}
Oct 31 01:11:07.436783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.436750    5419 generic.go:356] "Generic (PLEG): container finished" podID="40d1f3fb-ec45-43dd-b933-6367f1bc93a9" containerID="5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0" exitCode=0
Oct 31 01:11:07.436911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.436835    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h" event={"ID":"40d1f3fb-ec45-43dd-b933-6367f1bc93a9","Type":"ContainerDied","Data":"5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0"}
Oct 31 01:11:07.436911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.436875    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h" event={"ID":"40d1f3fb-ec45-43dd-b933-6367f1bc93a9","Type":"ContainerDied","Data":"31ad891b770de793dc949306fae449f85f4271497beface1201fc3a617ee312a"}
Oct 31 01:11:07.436911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.436898    5419 scope.go:122] "RemoveContainer" containerID="5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0"
Oct 31 01:11:07.437304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.437281    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4pj7h"
Oct 31 01:11:07.440289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.440067    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5979fb47-wtnx9" event={"ID":"c88b8389-01e2-47a6-bda6-d50b957cd4cf","Type":"ContainerStarted","Data":"40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40"}
Oct 31 01:11:07.440289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.440196    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6d5979fb47-wtnx9" podUID="c88b8389-01e2-47a6-bda6-d50b957cd4cf" containerName="authorino" containerID="cri-o://40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40" gracePeriod=30
Oct 31 01:11:07.446656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.446624    5419 scope.go:122] "RemoveContainer" containerID="5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0"
Oct 31 01:11:07.446933 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:11:07.446914    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0\": container with ID starting with 5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0 not found: ID does not exist" containerID="5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0"
Oct 31 01:11:07.446986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.446943    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0"} err="failed to get container status \"5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0\": rpc error: code = NotFound desc = could not find container \"5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0\": container with ID starting with 5c0aa3255d5819c7311ee2bc5a2323684db8585f36c54776d68d71210bbd06a0 not found: ID does not exist"
Oct 31 01:11:07.550079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.549077    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57c84b77c7-h6nf2" podStartSLOduration=2.268492318 podStartE2EDuration="3.549058407s" totalImagesPullingTime="1.280566089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:07.548219921 +0000 UTC m=+388.849176499" watchObservedRunningTime="2026-10-31 01:11:07.549058407 +0000 UTC m=+388.850015181"
Oct 31 01:11:07.577831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.577775    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m94b9"]
Oct 31 01:11:07.578041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.578011    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9" podUID="2556438b-6208-4049-b72a-da6a62a82b14" containerName="authorino" containerID="cri-o://3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867" gracePeriod=30
Oct 31 01:11:07.657596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.657564    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4pj7h"]
Oct 31 01:11:07.661555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.661527    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4pj7h"]
Oct 31 01:11:07.714441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.714410    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:07.774618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.774553    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d5979fb47-wtnx9" podStartSLOduration=1.968851334 podStartE2EDuration="3.774534597s" totalImagesPullingTime="1.805683263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:07.77347989 +0000 UTC m=+389.074436485" watchObservedRunningTime="2026-10-31 01:11:07.774534597 +0000 UTC m=+389.075491178"
Oct 31 01:11:07.804447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.804371    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9\" (UniqueName: \"kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9\") pod \"c88b8389-01e2-47a6-bda6-d50b957cd4cf\" (UID: \"c88b8389-01e2-47a6-bda6-d50b957cd4cf\") "
Oct 31 01:11:07.807609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.807572    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9" pod "c88b8389-01e2-47a6-bda6-d50b957cd4cf" (UID: "c88b8389-01e2-47a6-bda6-d50b957cd4cf"). InnerVolumeSpecName "kube-api-access-5sck9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:07.826706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.826686    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:11:07.905118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.905089    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh\" (UniqueName: \"kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh\") pod \"2556438b-6208-4049-b72a-da6a62a82b14\" (UID: \"2556438b-6208-4049-b72a-da6a62a82b14\") "
Oct 31 01:11:07.905326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.905312    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sck9\" (UniqueName: \"kubernetes.io/projected/c88b8389-01e2-47a6-bda6-d50b957cd4cf-kube-api-access-5sck9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.907328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:07.907301    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh" pod "2556438b-6208-4049-b72a-da6a62a82b14" (UID: "2556438b-6208-4049-b72a-da6a62a82b14"). InnerVolumeSpecName "kube-api-access-dxjdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:08.005959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.005930    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjdh\" (UniqueName: \"kubernetes.io/projected/2556438b-6208-4049-b72a-da6a62a82b14-kube-api-access-dxjdh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:08.446543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.446507    5419 generic.go:356] "Generic (PLEG): container finished" podID="2556438b-6208-4049-b72a-da6a62a82b14" containerID="3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867" exitCode=0
Oct 31 01:11:08.447030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.446552    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9"
Oct 31 01:11:08.447030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.446591    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9" event={"ID":"2556438b-6208-4049-b72a-da6a62a82b14","Type":"ContainerDied","Data":"3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867"}
Oct 31 01:11:08.447030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.446637    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m94b9" event={"ID":"2556438b-6208-4049-b72a-da6a62a82b14","Type":"ContainerDied","Data":"b31455d28487c127c29a4138fbff96a5e575e2f18b2c614540fd1ad57e8c77a5"}
Oct 31 01:11:08.447030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.446658    5419 scope.go:122] "RemoveContainer" containerID="3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867"
Oct 31 01:11:08.448971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.448941    5419 generic.go:356] "Generic (PLEG): container finished" podID="c88b8389-01e2-47a6-bda6-d50b957cd4cf" containerID="40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40" exitCode=0
Oct 31 01:11:08.448971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.448973    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d5979fb47-wtnx9"
Oct 31 01:11:08.449180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.449000    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5979fb47-wtnx9" event={"ID":"c88b8389-01e2-47a6-bda6-d50b957cd4cf","Type":"ContainerDied","Data":"40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40"}
Oct 31 01:11:08.449180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.449032    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d5979fb47-wtnx9" event={"ID":"c88b8389-01e2-47a6-bda6-d50b957cd4cf","Type":"ContainerDied","Data":"0e9dce3845d95176bcd2f92b8f14f4a5f0ceb6aa8928d6e0b44bf1f5b5111cc1"}
Oct 31 01:11:08.454951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.454933    5419 scope.go:122] "RemoveContainer" containerID="3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867"
Oct 31 01:11:08.455236 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:11:08.455210    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867\": container with ID starting with 3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867 not found: ID does not exist" containerID="3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867"
Oct 31 01:11:08.455315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.455242    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867"} err="failed to get container status \"3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867\": rpc error: code = NotFound desc = could not find container \"3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867\": container with ID starting with 3cef562e3422f1658b2fb8fead0c4b3c0a1a09edf565e3e5984c2bd7e1395867 not found: ID does not exist"
Oct 31 01:11:08.455315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.455264    5419 scope.go:122] "RemoveContainer" containerID="40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40"
Oct 31 01:11:08.462822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.462805    5419 scope.go:122] "RemoveContainer" containerID="40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40"
Oct 31 01:11:08.463090 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:11:08.463074    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40\": container with ID starting with 40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40 not found: ID does not exist" containerID="40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40"
Oct 31 01:11:08.463129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.463098    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40"} err="failed to get container status \"40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40\": rpc error: code = NotFound desc = could not find container \"40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40\": container with ID starting with 40f473510d5500bcc6baa6f129fbb0ccbfdde92cdaf72d7aefd4cc833916fa40 not found: ID does not exist"
Oct 31 01:11:08.606058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.606023    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m94b9"]
Oct 31 01:11:08.607935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.607914    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m94b9"]
Oct 31 01:11:08.715063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.714976    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d5979fb47-wtnx9"]
Oct 31 01:11:08.718629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.718601    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d5979fb47-wtnx9"]
Oct 31 01:11:08.834720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.834692    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2556438b-6208-4049-b72a-da6a62a82b14" path="/var/lib/kubelet/pods/2556438b-6208-4049-b72a-da6a62a82b14/volumes"
Oct 31 01:11:08.834998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.834986    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d1f3fb-ec45-43dd-b933-6367f1bc93a9" path="/var/lib/kubelet/pods/40d1f3fb-ec45-43dd-b933-6367f1bc93a9/volumes"
Oct 31 01:11:08.835282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:11:08.835270    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88b8389-01e2-47a6-bda6-d50b957cd4cf" path="/var/lib/kubelet/pods/c88b8389-01e2-47a6-bda6-d50b957cd4cf/volumes"
Oct 31 01:11:14.580941 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:11:14.580902    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:11:44.580560 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:11:44.580521    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:12:06.502712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.502676    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"]
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.502980    5419 cpu_manager.go:414] "containerMap: removing container" podUID="40d1f3fb-ec45-43dd-b933-6367f1bc93a9" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.502993    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d1f3fb-ec45-43dd-b933-6367f1bc93a9" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503019    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c88b8389-01e2-47a6-bda6-d50b957cd4cf" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503024    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88b8389-01e2-47a6-bda6-d50b957cd4cf" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503035    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2556438b-6208-4049-b72a-da6a62a82b14" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503040    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2556438b-6208-4049-b72a-da6a62a82b14" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503101    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2556438b-6208-4049-b72a-da6a62a82b14" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503109    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d1f3fb-ec45-43dd-b933-6367f1bc93a9" containerName="authorino"
Oct 31 01:12:06.503133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.503119    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88b8389-01e2-47a6-bda6-d50b957cd4cf" containerName="authorino"
Oct 31 01:12:06.505934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.505918    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.557958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.557921    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 01:12:06.566708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.566673    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3606b3-e3d1-4bd8-bf2e-424c31fe3756-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rsgqk\" (UID: \"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.566840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.566806    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gddh\" (UniqueName: \"kubernetes.io/projected/1c3606b3-e3d1-4bd8-bf2e-424c31fe3756-kube-api-access-8gddh\") pod \"payload-pre-processing-7b98bb678c-rsgqk\" (UID: \"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.607106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.607077    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xbgfw\""
Oct 31 01:12:06.608099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.608081    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"]
Oct 31 01:12:06.668083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.668050    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3606b3-e3d1-4bd8-bf2e-424c31fe3756-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rsgqk\" (UID: \"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.668255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.668108    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gddh\" (UniqueName: \"kubernetes.io/projected/1c3606b3-e3d1-4bd8-bf2e-424c31fe3756-kube-api-access-8gddh\") pod \"payload-pre-processing-7b98bb678c-rsgqk\" (UID: \"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.668748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.668728    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c3606b3-e3d1-4bd8-bf2e-424c31fe3756-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rsgqk\" (UID: \"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.725458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.725434    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gddh\" (UniqueName: \"kubernetes.io/projected/1c3606b3-e3d1-4bd8-bf2e-424c31fe3756-kube-api-access-8gddh\") pod \"payload-pre-processing-7b98bb678c-rsgqk\" (UID: \"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:06.817778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:06.817703    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:07.188523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:07.188445    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"]
Oct 31 01:12:07.191624 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:12:07.191577    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3606b3_e3d1_4bd8_bf2e_424c31fe3756.slice/crio-ef509adf04272dbea7547c3c736508e929b2be784bea15b7f292e5b3b0280c89 WatchSource:0}: Error finding container ef509adf04272dbea7547c3c736508e929b2be784bea15b7f292e5b3b0280c89: Status 404 returned error can't find the container with id ef509adf04272dbea7547c3c736508e929b2be784bea15b7f292e5b3b0280c89
Oct 31 01:12:07.690801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:07.690764    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk" event={"ID":"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756","Type":"ContainerStarted","Data":"ef509adf04272dbea7547c3c736508e929b2be784bea15b7f292e5b3b0280c89"}
Oct 31 01:12:10.704626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:10.704582    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk" event={"ID":"1c3606b3-e3d1-4bd8-bf2e-424c31fe3756","Type":"ContainerStarted","Data":"ca7e2bc015348031ec96ad0f46253867374abbb50520645082f0eec7ddea5076"}
Oct 31 01:12:10.705118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:10.704701    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:10.820956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:10.820905    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk" podStartSLOduration=2.108611982 podStartE2EDuration="4.820892791s" totalImagesPullingTime="2.712280809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:10.819511175 +0000 UTC m=+452.120467757" watchObservedRunningTime="2026-10-31 01:12:10.820892791 +0000 UTC m=+452.121849383"
Oct 31 01:12:14.580807 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:12:14.580768    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:12:21.712351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:21.712275    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rsgqk"
Oct 31 01:12:44.580698 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:12:44.580662    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:12:50.911014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.910984    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"]
Oct 31 01:12:50.912953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.912933    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.939593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.939565    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.939718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.939597    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.939777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.939710    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.939777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.939740    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqxff\" (UniqueName: \"kubernetes.io/projected/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-kube-api-access-hqxff\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.939870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.939778    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.939870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.939815    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:50.992593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:50.992562    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 01:12:51.014430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.014401    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-btdsg\""
Oct 31 01:12:51.014430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.014405    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 01:12:51.014578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.014435    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 01:12:51.015425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.015407    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"]
Oct 31 01:12:51.041175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041130    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041187    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqxff\" (UniqueName: \"kubernetes.io/projected/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-kube-api-access-hqxff\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041230    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041285    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041344    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041372    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041510    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041701    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.041719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.041713    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.043560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.043541    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.043808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.043791    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.098748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.098726    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqxff\" (UniqueName: \"kubernetes.io/projected/05fa0135-c2dc-48b5-80bb-0e48e9eca65c-kube-api-access-hqxff\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk\" (UID: \"05fa0135-c2dc-48b5-80bb-0e48e9eca65c\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.224186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.224159    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:51.595301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.595202    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"]
Oct 31 01:12:51.598156 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:12:51.598114    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fa0135_c2dc_48b5_80bb_0e48e9eca65c.slice/crio-7b947ea23a99ef95fcff83d6c96e8be346c2a4984c6c43cc7b12e485a74d1340 WatchSource:0}: Error finding container 7b947ea23a99ef95fcff83d6c96e8be346c2a4984c6c43cc7b12e485a74d1340: Status 404 returned error can't find the container with id 7b947ea23a99ef95fcff83d6c96e8be346c2a4984c6c43cc7b12e485a74d1340
Oct 31 01:12:51.869846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:51.869769    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk" event={"ID":"05fa0135-c2dc-48b5-80bb-0e48e9eca65c","Type":"ContainerStarted","Data":"7b947ea23a99ef95fcff83d6c96e8be346c2a4984c6c43cc7b12e485a74d1340"}
Oct 31 01:12:53.883080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:53.883040    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk" event={"ID":"05fa0135-c2dc-48b5-80bb-0e48e9eca65c","Type":"ContainerStarted","Data":"470aec7dfe915ebe36769eb62cf779c73611bc6ea0446003f6753b3676bde6e1"}
Oct 31 01:12:53.998989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:53.998937    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk" podStartSLOduration=2.569435264 podStartE2EDuration="3.998922696s" totalImagesPullingTime="1.429487432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:53.997571152 +0000 UTC m=+495.298527731" watchObservedRunningTime="2026-10-31 01:12:53.998922696 +0000 UTC m=+495.299879270"
Oct 31 01:12:54.225131 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:54.225102    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:12:54.600762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:12:54.600679    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:13:01.225230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.225191    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:13:01.237604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.237583    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:13:01.602007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.601927    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"]
Oct 31 01:13:01.604033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.604018    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.655648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.655613    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 01:13:01.663198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.663169    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"]
Oct 31 01:13:01.742833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.742797    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87dac0d-52ca-4169-986f-95ea2aa75916-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.743003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.742839    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.743003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.742921    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.743003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.742958    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.743003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.742981    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4xn\" (UniqueName: \"kubernetes.io/projected/e87dac0d-52ca-4169-986f-95ea2aa75916-kube-api-access-6x4xn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.743159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.743098    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.844426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.844389    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.844426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.844433    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87dac0d-52ca-4169-986f-95ea2aa75916-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.844691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.844564    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.844691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.844633    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.844691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.844677    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.844849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.844711    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4xn\" (UniqueName: \"kubernetes.io/projected/e87dac0d-52ca-4169-986f-95ea2aa75916-kube-api-access-6x4xn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.845028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.845003    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.845185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.845042    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.845185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.845171    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.846923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.846904    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87dac0d-52ca-4169-986f-95ea2aa75916-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.847104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.847088    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87dac0d-52ca-4169-986f-95ea2aa75916-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.901706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.901631    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4xn\" (UniqueName: \"kubernetes.io/projected/e87dac0d-52ca-4169-986f-95ea2aa75916-kube-api-access-6x4xn\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7rlk4\" (UID: \"e87dac0d-52ca-4169-986f-95ea2aa75916\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.914962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.914927    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:01.927094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:01.927066    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk"
Oct 31 01:13:02.074021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:02.073986    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:13:02.290855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:02.290822    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"]
Oct 31 01:13:02.293994 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:13:02.293967    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87dac0d_52ca_4169_986f_95ea2aa75916.slice/crio-4d566808cde40d6795606eafa1d26ebe34b66231ee7c859a2250dbc393a037c3 WatchSource:0}: Error finding container 4d566808cde40d6795606eafa1d26ebe34b66231ee7c859a2250dbc393a037c3: Status 404 returned error can't find the container with id 4d566808cde40d6795606eafa1d26ebe34b66231ee7c859a2250dbc393a037c3
Oct 31 01:13:02.920513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:02.920488    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4" event={"ID":"e87dac0d-52ca-4169-986f-95ea2aa75916","Type":"ContainerStarted","Data":"4d566808cde40d6795606eafa1d26ebe34b66231ee7c859a2250dbc393a037c3"}
Oct 31 01:13:03.925936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:03.925906    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4" event={"ID":"e87dac0d-52ca-4169-986f-95ea2aa75916","Type":"ContainerStarted","Data":"57b641f944825eed57da585a17b89b8bb97742c249eba29db941ed59dd581278"}
Oct 31 01:13:04.042303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:04.042249    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4" podStartSLOduration=2.469425525 podStartE2EDuration="3.042232826s" totalImagesPullingTime="572.807301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:04.041066973 +0000 UTC m=+505.342023550" watchObservedRunningTime="2026-10-31 01:13:04.042232826 +0000 UTC m=+505.343189402"
Oct 31 01:13:04.915825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:04.915792    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:09.240422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.240388    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"]
Oct 31 01:13:09.242900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.242879    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.302855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.302808    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qttlc\" (UniqueName: \"kubernetes.io/projected/78e23333-17c5-4811-ade2-829d0e2d5548-kube-api-access-qttlc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.303038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.302863    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.303110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.303043    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.303196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.303106    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.303196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.303176    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e23333-17c5-4811-ade2-829d0e2d5548-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.303267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.303242    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.343716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.343690    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 01:13:09.344840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.344822    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"]
Oct 31 01:13:09.404094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404056    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404097    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404115    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e23333-17c5-4811-ade2-829d0e2d5548-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404282    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404354    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qttlc\" (UniqueName: \"kubernetes.io/projected/78e23333-17c5-4811-ade2-829d0e2d5548-kube-api-access-qttlc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404382    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404672    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404709    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.404930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.404729    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.406537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.406516    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78e23333-17c5-4811-ade2-829d0e2d5548-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.406697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.406678    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78e23333-17c5-4811-ade2-829d0e2d5548-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.460819 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.460780    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qttlc\" (UniqueName: \"kubernetes.io/projected/78e23333-17c5-4811-ade2-829d0e2d5548-kube-api-access-qttlc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f\" (UID: \"78e23333-17c5-4811-ade2-829d0e2d5548\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.553687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.553600    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:09.921500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.921427    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"]
Oct 31 01:13:09.924716 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:13:09.924692    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e23333_17c5_4811_ade2_829d0e2d5548.slice/crio-d6a088747d317797bd504d8451849b60a9496e4e9917a826fe52a36efc3ff319 WatchSource:0}: Error finding container d6a088747d317797bd504d8451849b60a9496e4e9917a826fe52a36efc3ff319: Status 404 returned error can't find the container with id d6a088747d317797bd504d8451849b60a9496e4e9917a826fe52a36efc3ff319
Oct 31 01:13:09.964032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:09.964001    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f" event={"ID":"78e23333-17c5-4811-ade2-829d0e2d5548","Type":"ContainerStarted","Data":"d6a088747d317797bd504d8451849b60a9496e4e9917a826fe52a36efc3ff319"}
Oct 31 01:13:10.970006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:10.969968    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f" event={"ID":"78e23333-17c5-4811-ade2-829d0e2d5548","Type":"ContainerStarted","Data":"2a545ca3256f3096303cdc90e9da4b5931f698a6d3497daadf58abb2ff685e92"}
Oct 31 01:13:11.087195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:11.087121    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f" podStartSLOduration=1.5179524469999999 podStartE2EDuration="2.087102826s" totalImagesPullingTime="569.150379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:11.085930206 +0000 UTC m=+512.386886785" watchObservedRunningTime="2026-10-31 01:13:11.087102826 +0000 UTC m=+512.388059420"
Oct 31 01:13:11.553999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:11.553963    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:11.915638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:11.915565    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:11.928045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:11.928023    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:11.984787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:11.984764    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7rlk4"
Oct 31 01:13:14.581209 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:13:14.581179    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:13:16.616159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:16.615970    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:13:19.554732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:19.554695    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:19.567058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:19.567036    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:20.016920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.016893    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f"
Oct 31 01:13:20.445717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.445637    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"]
Oct 31 01:13:20.447733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.447717    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.500199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.500171    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 01:13:20.550120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.550082    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"]
Oct 31 01:13:20.600723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.600687    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.601242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.600731    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrss5\" (UniqueName: \"kubernetes.io/projected/222d5179-79a4-4cfd-b925-6a899bb608ba-kube-api-access-vrss5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.601242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.600755    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.601242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.600945    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.601242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.600983    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222d5179-79a4-4cfd-b925-6a899bb608ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.601242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.601094    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.701859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.701778    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.701859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.701811    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222d5179-79a4-4cfd-b925-6a899bb608ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.701859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.701842    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.702183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.701877    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.702183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.701922    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrss5\" (UniqueName: \"kubernetes.io/projected/222d5179-79a4-4cfd-b925-6a899bb608ba-kube-api-access-vrss5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.702183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.701996    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.702353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.702313    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.702423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.702400    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.702461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.702442    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.704364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.704345    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/222d5179-79a4-4cfd-b925-6a899bb608ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.704597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.704581    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/222d5179-79a4-4cfd-b925-6a899bb608ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:20.759238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:20.759209    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrss5\" (UniqueName: \"kubernetes.io/projected/222d5179-79a4-4cfd-b925-6a899bb608ba-kube-api-access-vrss5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc\" (UID: \"222d5179-79a4-4cfd-b925-6a899bb608ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:21.057316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:21.057279    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:21.427945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:21.427922    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"]
Oct 31 01:13:21.430402 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:13:21.430371    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod222d5179_79a4_4cfd_b925_6a899bb608ba.slice/crio-99509c61af14c5a85dd20b07f3d14dd9b01c4dbd9334fb7ad4cf44d46b3b2af1 WatchSource:0}: Error finding container 99509c61af14c5a85dd20b07f3d14dd9b01c4dbd9334fb7ad4cf44d46b3b2af1: Status 404 returned error can't find the container with id 99509c61af14c5a85dd20b07f3d14dd9b01c4dbd9334fb7ad4cf44d46b3b2af1
Oct 31 01:13:22.014744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:22.014702    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" event={"ID":"222d5179-79a4-4cfd-b925-6a899bb608ba","Type":"ContainerStarted","Data":"99509c61af14c5a85dd20b07f3d14dd9b01c4dbd9334fb7ad4cf44d46b3b2af1"}
Oct 31 01:13:23.020587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:23.020554    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" event={"ID":"222d5179-79a4-4cfd-b925-6a899bb608ba","Type":"ContainerStarted","Data":"4b5211541d0b1d0195401f149af3c539549a1bbaec883d13bfb65f0e8311a6a0"}
Oct 31 01:13:23.058056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:23.058014    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:23.136620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:23.136565    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podStartSLOduration=2.475621997 podStartE2EDuration="3.136551461s" totalImagesPullingTime="660.929464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:23.135867237 +0000 UTC m=+524.436823839" watchObservedRunningTime="2026-10-31 01:13:23.136551461 +0000 UTC m=+524.437508039"
Oct 31 01:13:26.349962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.349924    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"]
Oct 31 01:13:26.353649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.353632    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.405999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.405965    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bnzrd\""
Oct 31 01:13:26.456696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.456665    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"]
Oct 31 01:13:26.464978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.464954    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.464983    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465008    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465031    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465221    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465255    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465274    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhwh\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465388    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.465465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.465431    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.565924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.565895    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.565937    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.565959    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhwh\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.565987    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566188    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566241    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566271    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566310    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566356    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566406    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566457    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566572    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.566649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.566633    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.567133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.567106    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.568439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.568421    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.568834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.568815    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.622852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.622795    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.623082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.623060    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhwh\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-trv8z\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:26.665764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:26.665736    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:27.089789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:27.089633    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"]
Oct 31 01:13:27.092285 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:13:27.092251    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652d52af_d44a_4878_82a1_3e57b0be0b16.slice/crio-7e904b10b885983b987f7049ac417f42ef1a52f6e345c70689889f48f9d29cfe WatchSource:0}: Error finding container 7e904b10b885983b987f7049ac417f42ef1a52f6e345c70689889f48f9d29cfe: Status 404 returned error can't find the container with id 7e904b10b885983b987f7049ac417f42ef1a52f6e345c70689889f48f9d29cfe
Oct 31 01:13:28.043233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:28.043199    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" event={"ID":"652d52af-d44a-4878-82a1-3e57b0be0b16","Type":"ContainerStarted","Data":"7e904b10b885983b987f7049ac417f42ef1a52f6e345c70689889f48f9d29cfe"}
Oct 31 01:13:30.218666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:30.218629    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:13:31.058390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:31.058358    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:31.073021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:31.072993    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:31.250400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:31.250366    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:13:31.250681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:31.250455    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:13:31.250681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:31.250501    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:13:32.063397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:32.063359    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" event={"ID":"652d52af-d44a-4878-82a1-3e57b0be0b16","Type":"ContainerStarted","Data":"fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6"}
Oct 31 01:13:32.069260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:32.069237    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:13:32.231703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:32.231639    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podStartSLOduration=2.076534179 podStartE2EDuration="6.231625106s" totalImagesPullingTime="4.155090927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:32.23053451 +0000 UTC m=+533.531491087" watchObservedRunningTime="2026-10-31 01:13:32.231625106 +0000 UTC m=+533.532581683"
Oct 31 01:13:32.666196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:32.666161    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:32.667719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:32.667685    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:32.667840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:32.667742    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:33.666858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:33.666821    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:33.667338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:33.666883    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:34.666308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:34.666268    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:34.666493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:34.666326    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:35.666216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:35.666180    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:35.666622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:35.666234    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:36.667030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:36.666992    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:36.667432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:36.667047    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:37.666688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:37.666643    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:37.666913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:37.666695    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:38.666202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:38.666159    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 31 01:13:38.666715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:38.666211    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 31 01:13:39.670362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:39.670329    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:39.670756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:39.670626    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:39.671402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:39.671385    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:13:44.581215 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:13:44.581178    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:13:48.662464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.662372    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65fdc4ffcd-ncpr6"]
Oct 31 01:13:48.666059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.666035    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.722537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.722507    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65fdc4ffcd-ncpr6"]
Oct 31 01:13:48.752389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.752360    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27gk\" (UniqueName: \"kubernetes.io/projected/779ab764-2e40-47e1-b415-0f7307c9d002-kube-api-access-c27gk\") pod \"authorino-65fdc4ffcd-ncpr6\" (UID: \"779ab764-2e40-47e1-b415-0f7307c9d002\") " pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.752547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.752414    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/779ab764-2e40-47e1-b415-0f7307c9d002-tls-cert\") pod \"authorino-65fdc4ffcd-ncpr6\" (UID: \"779ab764-2e40-47e1-b415-0f7307c9d002\") " pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.853050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.853013    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27gk\" (UniqueName: \"kubernetes.io/projected/779ab764-2e40-47e1-b415-0f7307c9d002-kube-api-access-c27gk\") pod \"authorino-65fdc4ffcd-ncpr6\" (UID: \"779ab764-2e40-47e1-b415-0f7307c9d002\") " pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.853050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.853062    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/779ab764-2e40-47e1-b415-0f7307c9d002-tls-cert\") pod \"authorino-65fdc4ffcd-ncpr6\" (UID: \"779ab764-2e40-47e1-b415-0f7307c9d002\") " pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.855585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.855562    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/779ab764-2e40-47e1-b415-0f7307c9d002-tls-cert\") pod \"authorino-65fdc4ffcd-ncpr6\" (UID: \"779ab764-2e40-47e1-b415-0f7307c9d002\") " pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.909815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.909777    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27gk\" (UniqueName: \"kubernetes.io/projected/779ab764-2e40-47e1-b415-0f7307c9d002-kube-api-access-c27gk\") pod \"authorino-65fdc4ffcd-ncpr6\" (UID: \"779ab764-2e40-47e1-b415-0f7307c9d002\") " pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:48.977092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:48.977062    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6"
Oct 31 01:13:49.343805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:49.343716    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65fdc4ffcd-ncpr6"]
Oct 31 01:13:49.346771 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:13:49.346734    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod779ab764_2e40_47e1_b415_0f7307c9d002.slice/crio-4c0c0964eec7999b851f99e413fd9e71cb8390a304528a24d921f26ccc1da6c8 WatchSource:0}: Error finding container 4c0c0964eec7999b851f99e413fd9e71cb8390a304528a24d921f26ccc1da6c8: Status 404 returned error can't find the container with id 4c0c0964eec7999b851f99e413fd9e71cb8390a304528a24d921f26ccc1da6c8
Oct 31 01:13:50.162376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:50.162340    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6" event={"ID":"779ab764-2e40-47e1-b415-0f7307c9d002","Type":"ContainerStarted","Data":"4c0c0964eec7999b851f99e413fd9e71cb8390a304528a24d921f26ccc1da6c8"}
Oct 31 01:13:51.167889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.167803    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6" event={"ID":"779ab764-2e40-47e1-b415-0f7307c9d002","Type":"ContainerStarted","Data":"fb7e3cca41a83382f2b0f88df2c4f5619858ea69640f12773bac9c3c8d9517ac"}
Oct 31 01:13:51.285510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.285462    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65fdc4ffcd-ncpr6" podStartSLOduration=1.735781134 podStartE2EDuration="3.285448349s" totalImagesPullingTime="1.549667215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:51.28522905 +0000 UTC m=+552.586185627" watchObservedRunningTime="2026-10-31 01:13:51.285448349 +0000 UTC m=+552.586404960"
Oct 31 01:13:51.335720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.335680    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57c84b77c7-h6nf2"]
Oct 31 01:13:51.335934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.335911    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57c84b77c7-h6nf2" podUID="cf8c0fcb-20b9-4016-aebd-637adc253c6c" containerName="authorino" containerID="cri-o://10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c" gracePeriod=30
Oct 31 01:13:51.585051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.585025    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:13:51.678462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.678416    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5\" (UniqueName: \"kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5\") pod \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") "
Oct 31 01:13:51.678637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.678552    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert\") pod \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\" (UID: \"cf8c0fcb-20b9-4016-aebd-637adc253c6c\") "
Oct 31 01:13:51.680691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.680658    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5" pod "cf8c0fcb-20b9-4016-aebd-637adc253c6c" (UID: "cf8c0fcb-20b9-4016-aebd-637adc253c6c"). InnerVolumeSpecName "kube-api-access-tlkd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:51.688508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.688482    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert" pod "cf8c0fcb-20b9-4016-aebd-637adc253c6c" (UID: "cf8c0fcb-20b9-4016-aebd-637adc253c6c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:13:51.779508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.779472    5419 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cf8c0fcb-20b9-4016-aebd-637adc253c6c-tls-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:51.779508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:51.779505    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlkd5\" (UniqueName: \"kubernetes.io/projected/cf8c0fcb-20b9-4016-aebd-637adc253c6c-kube-api-access-tlkd5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:52.174288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.174197    5419 generic.go:356] "Generic (PLEG): container finished" podID="cf8c0fcb-20b9-4016-aebd-637adc253c6c" containerID="10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c" exitCode=0
Oct 31 01:13:52.174288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.174250    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c84b77c7-h6nf2"
Oct 31 01:13:52.174822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.174284    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c84b77c7-h6nf2" event={"ID":"cf8c0fcb-20b9-4016-aebd-637adc253c6c","Type":"ContainerDied","Data":"10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c"}
Oct 31 01:13:52.174822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.174332    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c84b77c7-h6nf2" event={"ID":"cf8c0fcb-20b9-4016-aebd-637adc253c6c","Type":"ContainerDied","Data":"4085ef869deebff3b23c19009f101d5e5b46b4395d6634a8af3675cbc44b8627"}
Oct 31 01:13:52.174822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.174343    5419 scope.go:122] "RemoveContainer" containerID="10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c"
Oct 31 01:13:52.187110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.187074    5419 scope.go:122] "RemoveContainer" containerID="10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c"
Oct 31 01:13:52.187424 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:13:52.187406    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c\": container with ID starting with 10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c not found: ID does not exist" containerID="10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c"
Oct 31 01:13:52.187479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.187434    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c"} err="failed to get container status \"10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c\": rpc error: code = NotFound desc = could not find container \"10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c\": container with ID starting with 10aec502b59e9b55bb9884546730d3f41cff1df7c3a63ebcd57d639379e1624c not found: ID does not exist"
Oct 31 01:13:52.335941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.335900    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57c84b77c7-h6nf2"]
Oct 31 01:13:52.384396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.384363    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57c84b77c7-h6nf2"]
Oct 31 01:13:52.836361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:13:52.836325    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8c0fcb-20b9-4016-aebd-637adc253c6c" path="/var/lib/kubelet/pods/cf8c0fcb-20b9-4016-aebd-637adc253c6c/volumes"
Oct 31 01:14:14.580968 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:14:14.580930    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:14:40.789817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.789782    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"]
Oct 31 01:14:40.790258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.790158    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cf8c0fcb-20b9-4016-aebd-637adc253c6c" containerName="authorino"
Oct 31 01:14:40.790258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.790170    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8c0fcb-20b9-4016-aebd-637adc253c6c" containerName="authorino"
Oct 31 01:14:40.790258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.790241    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8c0fcb-20b9-4016-aebd-637adc253c6c" containerName="authorino"
Oct 31 01:14:40.794465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.794444    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.821738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821697    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.821927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821757    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kl7\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.821927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821851    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.821927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821885    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.821927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821922    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.822119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821944    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.822119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.821966    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.822119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.822054    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.822305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.822164    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.892760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.892728    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"]
Oct 31 01:14:40.920690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.920664    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f94871a1-openshift-default-dockercfg-vz7cd\""
Oct 31 01:14:40.923063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923044    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923083    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kl7\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923216    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923270    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923303    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923332    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923370    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923593    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923621    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923716    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923719    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923779    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923821    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.923948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.923931    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.925839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.925818    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.926344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.926324    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.982311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.982270    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:40.982311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:40.982312    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kl7\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7\") pod \"e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") " pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:41.108572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:41.108489    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:41.532500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:41.532475    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"]
Oct 31 01:14:41.535009 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:41.534982    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00630b5c_589c_4b00_b8c2_e4055c355c9f.slice/crio-321dc656c3a899ec639e9111b1a4eb3348853515b122337e1f29c9d6ee9f481a WatchSource:0}: Error finding container 321dc656c3a899ec639e9111b1a4eb3348853515b122337e1f29c9d6ee9f481a: Status 404 returned error can't find the container with id 321dc656c3a899ec639e9111b1a4eb3348853515b122337e1f29c9d6ee9f481a
Oct 31 01:14:41.537323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:41.537291    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:41.537430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:41.537367    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:41.537430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:41.537411    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:42.399494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.399455    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" event={"ID":"00630b5c-589c-4b00-b8c2-e4055c355c9f","Type":"ContainerStarted","Data":"d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"}
Oct 31 01:14:42.399892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.399518    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" event={"ID":"00630b5c-589c-4b00-b8c2-e4055c355c9f","Type":"ContainerStarted","Data":"321dc656c3a899ec639e9111b1a4eb3348853515b122337e1f29c9d6ee9f481a"}
Oct 31 01:14:42.585105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.585049    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podStartSLOduration=2.585036541 podStartE2EDuration="2.585036541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:42.584827192 +0000 UTC m=+603.885783767" watchObservedRunningTime="2026-10-31 01:14:42.585036541 +0000 UTC m=+603.885993117"
Oct 31 01:14:42.662454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.662370    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"]
Oct 31 01:14:42.665901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.665880    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.718579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.718541    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7164dd00-openshift-default-dockercfg-ntzff\""
Oct 31 01:14:42.726434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.726403    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"]
Oct 31 01:14:42.744307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744278    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76rj\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744313    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744331    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744474    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744532    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744553    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744595    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744706    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.744834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.744814    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.845621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845587    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.845820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845633    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.845820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845678    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.845820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845726    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.845820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845763    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.845820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845802    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76rj\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845825    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845850    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.845914    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.846118    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.846193    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.846342    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.846378    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.846690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.846673    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.848278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.848250    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.848461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.848445    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.903375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.903344    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.903532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.903505    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76rj\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj\") pod \"e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") " pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:42.979304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:42.979273    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:43.109613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.109584    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:43.114348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.114325    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:43.403331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.403254    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"]
Oct 31 01:14:43.405343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.405321    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:43.406227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.406201    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:14:43.406893 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:43.406876    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ca9a39_e693_418a_b9ed_5f64ac4475b4.slice/crio-c907f0e22d9d4e4c6a55f473dd26b8eda23cd84daeb61bc0cd31a0ba6347123d WatchSource:0}: Error finding container c907f0e22d9d4e4c6a55f473dd26b8eda23cd84daeb61bc0cd31a0ba6347123d: Status 404 returned error can't find the container with id c907f0e22d9d4e4c6a55f473dd26b8eda23cd84daeb61bc0cd31a0ba6347123d
Oct 31 01:14:43.409533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.409504    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:43.409626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.409592    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:43.409684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:43.409642    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:44.410806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:44.410757    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" event={"ID":"63ca9a39-e693-418a-b9ed-5f64ac4475b4","Type":"ContainerStarted","Data":"9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"}
Oct 31 01:14:44.411272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:44.410829    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" event={"ID":"63ca9a39-e693-418a-b9ed-5f64ac4475b4","Type":"ContainerStarted","Data":"c907f0e22d9d4e4c6a55f473dd26b8eda23cd84daeb61bc0cd31a0ba6347123d"}
Oct 31 01:14:44.531599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:44.531545    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podStartSLOduration=2.531531314 podStartE2EDuration="2.531531314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:44.530544462 +0000 UTC m=+605.831501057" watchObservedRunningTime="2026-10-31 01:14:44.531531314 +0000 UTC m=+605.832487891"
Oct 31 01:14:44.581008 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:14:44.580977    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:14:44.980016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:44.979985    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:44.985174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:44.985152    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:45.414952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:45.414875    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:45.415791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:45.415770    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:14:48.784741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.784708    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"]
Oct 31 01:14:48.788307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.788291    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:48.847302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.847269    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a0c7a371\""
Oct 31 01:14:48.893674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.893638    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a0c7a371-dockercfg-vz4zp\""
Oct 31 01:14:48.895146 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.895113    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"]
Oct 31 01:14:48.900719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.900698    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:48.900782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:48.900737    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccpm\" (UniqueName: \"kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm\") pod \"payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:49.002100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:49.002065    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:49.002299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:49.002110    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccpm\" (UniqueName: \"kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm\") pod \"payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:49.002758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:49.002738    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:49.059317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:49.059250    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccpm\" (UniqueName: \"kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm\") pod \"payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:49.100385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:49.100358    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:49.472321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:49.472292    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"]
Oct 31 01:14:49.481582 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:49.475653    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd232b064_7f45_4696_9eab_5547972d6c1c.slice/crio-9a2fb1d324184258f3de2e27e44f0ebd76ccca2b68c46d55b0b3aa3687e2ac48 WatchSource:0}: Error finding container 9a2fb1d324184258f3de2e27e44f0ebd76ccca2b68c46d55b0b3aa3687e2ac48: Status 404 returned error can't find the container with id 9a2fb1d324184258f3de2e27e44f0ebd76ccca2b68c46d55b0b3aa3687e2ac48
Oct 31 01:14:50.455445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:50.455406    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" event={"ID":"d232b064-7f45-4696-9eab-5547972d6c1c","Type":"ContainerStarted","Data":"0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc"}
Oct 31 01:14:50.456154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:50.456069    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" event={"ID":"d232b064-7f45-4696-9eab-5547972d6c1c","Type":"ContainerStarted","Data":"9a2fb1d324184258f3de2e27e44f0ebd76ccca2b68c46d55b0b3aa3687e2ac48"}
Oct 31 01:14:50.456334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:50.456312    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:14:50.571693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:50.571637    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" podStartSLOduration=2.571624797 podStartE2EDuration="2.571624797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:50.570286914 +0000 UTC m=+611.871243494" watchObservedRunningTime="2026-10-31 01:14:50.571624797 +0000 UTC m=+611.872581373"
Oct 31 01:14:50.976556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:50.976524    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"]
Oct 31 01:14:50.979968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:50.979942    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.018864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.018832    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.018879    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019034    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019072    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019105    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019222    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbx9\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019260    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019311    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.019437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.019412    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.042738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.042709    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"]
Oct 31 01:14:51.081028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.080993    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-870eefeb-openshift-default-dockercfg-n7srj\""
Oct 31 01:14:51.120661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.120625    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121519    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121556    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121447    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121592    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbx9\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121934    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121960    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.121995    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.122048    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.122077    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.122163    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.122194    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.122495    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.123220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.122768    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.128756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.128721    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.128899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.128880    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.183244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.183215    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.183418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.183301    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbx9\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9\") pod \"e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") " pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.292301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.292211    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:51.720441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.720413    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"]
Oct 31 01:14:51.760479 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:51.760427    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26586095_56c1_4668_a769_0b96f2aecf4f.slice/crio-a8b070d82df52a74ae4dfa66d9a593854bcff4d23534df7b5da48e223feba2bc WatchSource:0}: Error finding container a8b070d82df52a74ae4dfa66d9a593854bcff4d23534df7b5da48e223feba2bc: Status 404 returned error can't find the container with id a8b070d82df52a74ae4dfa66d9a593854bcff4d23534df7b5da48e223feba2bc
Oct 31 01:14:51.763661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.763623    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:51.763763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.763718    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:51.763814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:51.763770    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:14:52.465880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:52.465843    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" event={"ID":"26586095-56c1-4668-a769-0b96f2aecf4f","Type":"ContainerStarted","Data":"ae3a066f7246a45efc970e4a7104cb054627800a214933a43ad4b69f187aa2d0"}
Oct 31 01:14:52.466062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:52.465895    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" event={"ID":"26586095-56c1-4668-a769-0b96f2aecf4f","Type":"ContainerStarted","Data":"a8b070d82df52a74ae4dfa66d9a593854bcff4d23534df7b5da48e223feba2bc"}
Oct 31 01:14:52.587404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:52.587353    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podStartSLOduration=2.58733749 podStartE2EDuration="2.58733749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:52.585272952 +0000 UTC m=+613.886229529" watchObservedRunningTime="2026-10-31 01:14:52.58733749 +0000 UTC m=+613.888294057"
Oct 31 01:14:53.293414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.293378    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:53.298602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.298579    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:53.470606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.470572    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:53.471507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.471490    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:14:53.526825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.526791    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"]
Oct 31 01:14:53.530313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.530291    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.582038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.581948    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f94871a1\""
Oct 31 01:14:53.582202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.582039    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f94871a1-dockercfg-kmvr6\""
Oct 31 01:14:53.600774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.600740    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"]
Oct 31 01:14:53.645437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.645397    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts4g\" (UniqueName: \"kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g\") pod \"payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.645714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.645681    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.681648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.681613    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"]
Oct 31 01:14:53.685720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.685697    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:53.743738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.743708    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"]
Oct 31 01:14:53.746268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.746241    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts4g\" (UniqueName: \"kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g\") pod \"payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.746400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.746275    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.746400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.746321    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume\") pod \"payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") " pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:53.746400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.746369    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvj2\" (UniqueName: \"kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2\") pod \"payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") " pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:53.747026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.746998    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.803176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.803126    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts4g\" (UniqueName: \"kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g\") pod \"payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.844218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.844117    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:53.847348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.847001    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvj2\" (UniqueName: \"kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2\") pod \"payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") " pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:53.847348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.847100    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume\") pod \"payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") " pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:53.847974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.847953    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume\") pod \"payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") " pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:53.906602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:53.906568    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvj2\" (UniqueName: \"kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2\") pod \"payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") " pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:54.002741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.002707    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:54.213745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.213669    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"]
Oct 31 01:14:54.217427 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:54.217398    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7382b896_814e_4467_9813_f78e881d0f85.slice/crio-014b6443ef7167a80ae331856c025f341036337dddb26bb5b03ec7b2fdb72906 WatchSource:0}: Error finding container 014b6443ef7167a80ae331856c025f341036337dddb26bb5b03ec7b2fdb72906: Status 404 returned error can't find the container with id 014b6443ef7167a80ae331856c025f341036337dddb26bb5b03ec7b2fdb72906
Oct 31 01:14:54.398278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.398246    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"]
Oct 31 01:14:54.401483 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:54.401450    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47700bc2_37d4_4dd0_8298_fb7ad293a978.slice/crio-5c7108e0acac639f473848ffc8fa59bc30366b5b3140b1636de733f24d0588d2 WatchSource:0}: Error finding container 5c7108e0acac639f473848ffc8fa59bc30366b5b3140b1636de733f24d0588d2: Status 404 returned error can't find the container with id 5c7108e0acac639f473848ffc8fa59bc30366b5b3140b1636de733f24d0588d2
Oct 31 01:14:54.485711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.485674    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q" event={"ID":"47700bc2-37d4-4dd0-8298-fb7ad293a978","Type":"ContainerStarted","Data":"5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672"}
Oct 31 01:14:54.485904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.485743    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q" event={"ID":"47700bc2-37d4-4dd0-8298-fb7ad293a978","Type":"ContainerStarted","Data":"5c7108e0acac639f473848ffc8fa59bc30366b5b3140b1636de733f24d0588d2"}
Oct 31 01:14:54.485986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.485924    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:14:54.488066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.488038    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg" event={"ID":"7382b896-814e-4467-9813-f78e881d0f85","Type":"ContainerStarted","Data":"11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b"}
Oct 31 01:14:54.488222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.488085    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg" event={"ID":"7382b896-814e-4467-9813-f78e881d0f85","Type":"ContainerStarted","Data":"014b6443ef7167a80ae331856c025f341036337dddb26bb5b03ec7b2fdb72906"}
Oct 31 01:14:54.605749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:54.605690    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q" podStartSLOduration=1.605677242 podStartE2EDuration="1.605677242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:54.605042643 +0000 UTC m=+615.905999221" watchObservedRunningTime="2026-10-31 01:14:54.605677242 +0000 UTC m=+615.906633819"
Oct 31 01:14:55.493004 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:55.492971    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:14:56.161921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.161864    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg" podStartSLOduration=3.161848404 podStartE2EDuration="3.161848404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:54.727230127 +0000 UTC m=+616.028186722" watchObservedRunningTime="2026-10-31 01:14:56.161848404 +0000 UTC m=+617.462804981"
Oct 31 01:14:56.162298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.162278    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"]
Oct 31 01:14:56.165742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.165721    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.225532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.225502    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"]
Oct 31 01:14:56.271360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.271321    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkdm\" (UniqueName: \"kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm\") pod \"payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") " pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.271525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.271380    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") " pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.272049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.272027    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7164dd00\""
Oct 31 01:14:56.272090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.272070    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7164dd00-dockercfg-4tgdb\""
Oct 31 01:14:56.372132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.372096    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") " pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.372323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.372214    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkdm\" (UniqueName: \"kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm\") pod \"payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") " pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.372863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.372846    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume\") pod \"payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") " pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.431941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.431865    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkdm\" (UniqueName: \"kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm\") pod \"payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") " pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.479008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.478983    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:56.860422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:56.860397    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"]
Oct 31 01:14:56.863230 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:14:56.863205    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45980ec5_7ede_4bfd_b21f_0edbcf91899f.slice/crio-fd484fe658a1730d831764e18c9c6db049c3339cd206a01e71d3dbf9ec3ee5ce WatchSource:0}: Error finding container fd484fe658a1730d831764e18c9c6db049c3339cd206a01e71d3dbf9ec3ee5ce: Status 404 returned error can't find the container with id fd484fe658a1730d831764e18c9c6db049c3339cd206a01e71d3dbf9ec3ee5ce
Oct 31 01:14:57.503362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:57.503330    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv" event={"ID":"45980ec5-7ede-4bfd-b21f-0edbcf91899f","Type":"ContainerStarted","Data":"1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8"}
Oct 31 01:14:57.503532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:57.503415    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv" event={"ID":"45980ec5-7ede-4bfd-b21f-0edbcf91899f","Type":"ContainerStarted","Data":"fd484fe658a1730d831764e18c9c6db049c3339cd206a01e71d3dbf9ec3ee5ce"}
Oct 31 01:14:57.503532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:57.503436    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:14:57.617763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:14:57.617710    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv" podStartSLOduration=1.617695943 podStartE2EDuration="1.617695943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:57.61692542 +0000 UTC m=+618.917882033" watchObservedRunningTime="2026-10-31 01:14:57.617695943 +0000 UTC m=+618.918652531"
Oct 31 01:15:00.168901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.168867    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"]
Oct 31 01:15:00.173001 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.172982    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"]
Oct 31 01:15:00.173127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.173112    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.176311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.176294    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.219452 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.219426    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-tml7w\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.219585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.219459    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.219585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.219492    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjgp\" (UniqueName: \"kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp\") pod \"maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.219681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.219636    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dkm\" (UniqueName: \"kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm\") pod \"maas-api-key-cleanup-29890155-tml7w\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.225176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.225154    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:15:00.225176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.225166    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:15:00.276328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.276299    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"]
Oct 31 01:15:00.277603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.277566    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"]
Oct 31 01:15:00.312986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.312958    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:15:00.320419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.320393    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-tml7w\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.320526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.320430    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.320526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.320462    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjgp\" (UniqueName: \"kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp\") pod \"maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.320526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.320483    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dkm\" (UniqueName: \"kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm\") pod \"maas-api-key-cleanup-29890155-tml7w\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.321056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.321036    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.321115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.321099    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890155-tml7w\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.376791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.376758    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjgp\" (UniqueName: \"kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp\") pod \"maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.376923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.376899    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dkm\" (UniqueName: \"kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm\") pod \"maas-api-key-cleanup-29890155-tml7w\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.486057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.486028    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:00.493276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.493256    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:00.882289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.882254    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"]
Oct 31 01:15:00.882722 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:00.882700    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a9919d_851e_4eca_bd12_6b1d6496ad63.slice/crio-02703394d63d55361943a57a8de1730228cc98dd7c8027507edfedbe45e4d38a WatchSource:0}: Error finding container 02703394d63d55361943a57a8de1730228cc98dd7c8027507edfedbe45e4d38a: Status 404 returned error can't find the container with id 02703394d63d55361943a57a8de1730228cc98dd7c8027507edfedbe45e4d38a
Oct 31 01:15:00.924612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:00.924577    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"]
Oct 31 01:15:00.927494 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:00.927467    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512fe42e_1236_4f12_9e3a_f7cec4cc8eab.slice/crio-6a9b545fd33ec6abff80251f4ec85eb31107958f13bd9ab93d25fb907847da44 WatchSource:0}: Error finding container 6a9b545fd33ec6abff80251f4ec85eb31107958f13bd9ab93d25fb907847da44: Status 404 returned error can't find the container with id 6a9b545fd33ec6abff80251f4ec85eb31107958f13bd9ab93d25fb907847da44
Oct 31 01:15:01.527477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:01.527440    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerStarted","Data":"02703394d63d55361943a57a8de1730228cc98dd7c8027507edfedbe45e4d38a"}
Oct 31 01:15:01.528587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:01.528561    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerStarted","Data":"6a9b545fd33ec6abff80251f4ec85eb31107958f13bd9ab93d25fb907847da44"}
Oct 31 01:15:02.468992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:02.468964    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:15:02.572486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:02.572452    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:15:03.318620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.318581    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"]
Oct 31 01:15:03.323832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.323808    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.387892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.386969    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"]
Oct 31 01:15:03.387892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.387850    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b9656f65-dockercfg-nssj5\""
Oct 31 01:15:03.433861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.433818    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b9656f65\""
Oct 31 01:15:03.462874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.462838    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5zm\" (UniqueName: \"kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm\") pod \"payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") " pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.463045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.463002    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume\") pod \"payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") " pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.564367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.564333    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5zm\" (UniqueName: \"kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm\") pod \"payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") " pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.564531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.564460    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume\") pod \"payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") " pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.565247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.565222    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume\") pod \"payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") " pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.623615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.623533    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5zm\" (UniqueName: \"kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm\") pod \"payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") " pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:03.646951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:03.646922    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:04.102337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:04.102306    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"]
Oct 31 01:15:04.105103 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:04.105073    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf219f56b_b1f7_4fa1_bf9c_cb95c2d5cb46.slice/crio-bd8a0e48ecc23ca87feedc1385e2d47651073fad981f192a1bdecdff25862d2a WatchSource:0}: Error finding container bd8a0e48ecc23ca87feedc1385e2d47651073fad981f192a1bdecdff25862d2a: Status 404 returned error can't find the container with id bd8a0e48ecc23ca87feedc1385e2d47651073fad981f192a1bdecdff25862d2a
Oct 31 01:15:04.547006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:04.546968    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9" event={"ID":"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46","Type":"ContainerStarted","Data":"02ec6b66a9874180a3b3a9e186ebf8b26cbe0e16f4a57124e28c0d178fbac068"}
Oct 31 01:15:04.547216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:04.547032    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9" event={"ID":"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46","Type":"ContainerStarted","Data":"bd8a0e48ecc23ca87feedc1385e2d47651073fad981f192a1bdecdff25862d2a"}
Oct 31 01:15:04.547216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:04.547060    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:04.665949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:04.665890    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9" podStartSLOduration=1.665877158 podStartE2EDuration="1.665877158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:04.665460676 +0000 UTC m=+625.966417257" watchObservedRunningTime="2026-10-31 01:15:04.665877158 +0000 UTC m=+625.966833735"
Oct 31 01:15:05.501283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:05.501254    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:15:06.502812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:06.502781    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:15:06.558128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:06.558038    5419 generic.go:356] "Generic (PLEG): container finished" podID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerID="0e74927368105787867c5131605ddbfceb122c308e7b2d146fa7ad41db058fc9" exitCode=60
Oct 31 01:15:06.558301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:06.558128    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerDied","Data":"0e74927368105787867c5131605ddbfceb122c308e7b2d146fa7ad41db058fc9"}
Oct 31 01:15:06.558394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:06.558374    5419 scope.go:122] "RemoveContainer" containerID="0e74927368105787867c5131605ddbfceb122c308e7b2d146fa7ad41db058fc9"
Oct 31 01:15:06.559840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:06.559818    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerStarted","Data":"8418048c7def218f7cfcca148e2f62afe8bb716b0d09462fb380c885666f74f5"}
Oct 31 01:15:06.734443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:06.734392    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" podStartSLOduration=1.8647301569999999 podStartE2EDuration="6.734378756s" totalImagesPullingTime="4.869648599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:06.733410205 +0000 UTC m=+628.034366773" watchObservedRunningTime="2026-10-31 01:15:06.734378756 +0000 UTC m=+628.035335332"
Oct 31 01:15:07.566323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:07.566284    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerStarted","Data":"3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a"}
Oct 31 01:15:07.567830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:07.567807    5419 generic.go:356] "Generic (PLEG): container finished" podID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerID="8418048c7def218f7cfcca148e2f62afe8bb716b0d09462fb380c885666f74f5" exitCode=60
Oct 31 01:15:07.567929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:07.567856    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerDied","Data":"8418048c7def218f7cfcca148e2f62afe8bb716b0d09462fb380c885666f74f5"}
Oct 31 01:15:07.568059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:07.568046    5419 scope.go:122] "RemoveContainer" containerID="8418048c7def218f7cfcca148e2f62afe8bb716b0d09462fb380c885666f74f5"
Oct 31 01:15:07.819031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:07.818904    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" podStartSLOduration=3.435650645 podStartE2EDuration="7.818885705s" totalImagesPullingTime="4.38323506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:07.702407475 +0000 UTC m=+629.003364054" watchObservedRunningTime="2026-10-31 01:15:07.818885705 +0000 UTC m=+629.119842284"
Oct 31 01:15:08.096218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.095506    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"]
Oct 31 01:15:08.102642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.101527    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.159872 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.159837    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-870eefeb\""
Oct 31 01:15:08.162563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.160459    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-870eefeb-dockercfg-pzzxf\""
Oct 31 01:15:08.164265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.164239    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"]
Oct 31 01:15:08.232520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.232485    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"]
Oct 31 01:15:08.235907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.235891    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.251950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.251926    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqgb\" (UniqueName: \"kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb\") pod \"payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.252054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.251983    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdmj\" (UniqueName: \"kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj\") pod \"payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") " pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.252111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.252090    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume\") pod \"payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") " pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.252177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.252159    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.299853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.299642    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"]
Oct 31 01:15:08.353354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.353262    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdmj\" (UniqueName: \"kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj\") pod \"payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") " pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.353354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.353328    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume\") pod \"payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") " pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.353568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.353371    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.353568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.353445    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqgb\" (UniqueName: \"kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb\") pod \"payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.354101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.354078    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.354187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.354084    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume\") pod \"payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") " pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.410501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.410461    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqgb\" (UniqueName: \"kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb\") pod \"payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.410635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.410510    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdmj\" (UniqueName: \"kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj\") pod \"payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") " pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.416397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.416376    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:08.511818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.511789    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:15:08.548902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.548875    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:08.599166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.597821    5419 generic.go:356] "Generic (PLEG): container finished" podID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerID="3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a" exitCode=60
Oct 31 01:15:08.599166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.598006    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerDied","Data":"3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a"}
Oct 31 01:15:08.599166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.598064    5419 scope.go:122] "RemoveContainer" containerID="0e74927368105787867c5131605ddbfceb122c308e7b2d146fa7ad41db058fc9"
Oct 31 01:15:08.599166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.598547    5419 scope.go:122] "RemoveContainer" containerID="3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a"
Oct 31 01:15:08.599166 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:08.598818    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb_odh-ai-gateway-infra(63a9919d-851e-4eca-bd12-6b1d6496ad63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63"
Oct 31 01:15:08.611543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.610884    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerStarted","Data":"41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac"}
Oct 31 01:15:08.786360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.786331    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"]
Oct 31 01:15:08.788381 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:08.788354    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafaf2744_8709_4540_962b_a5de808bc38a.slice/crio-7ce6fe8f4a00c286a5444ade84276e06e8dd6486221e921449c32322dcc1da57 WatchSource:0}: Error finding container 7ce6fe8f4a00c286a5444ade84276e06e8dd6486221e921449c32322dcc1da57: Status 404 returned error can't find the container with id 7ce6fe8f4a00c286a5444ade84276e06e8dd6486221e921449c32322dcc1da57
Oct 31 01:15:08.946289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:08.946262    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"]
Oct 31 01:15:08.947673 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:08.947646    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1215c783_ea3f_48af_b15d_5494467badca.slice/crio-d3712fa6aaea961e88f4f237836bdb822035b2ad71cec035cb0daf38cf55c830 WatchSource:0}: Error finding container d3712fa6aaea961e88f4f237836bdb822035b2ad71cec035cb0daf38cf55c830: Status 404 returned error can't find the container with id d3712fa6aaea961e88f4f237836bdb822035b2ad71cec035cb0daf38cf55c830
Oct 31 01:15:09.617478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.617434    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf" event={"ID":"1215c783-ea3f-48af-b15d-5494467badca","Type":"ContainerStarted","Data":"5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f"}
Oct 31 01:15:09.617955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.617547    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:09.617955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.617573    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf" event={"ID":"1215c783-ea3f-48af-b15d-5494467badca","Type":"ContainerStarted","Data":"d3712fa6aaea961e88f4f237836bdb822035b2ad71cec035cb0daf38cf55c830"}
Oct 31 01:15:09.619289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.619264    5419 generic.go:356] "Generic (PLEG): container finished" podID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerID="41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac" exitCode=60
Oct 31 01:15:09.619421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.619349    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerDied","Data":"41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac"}
Oct 31 01:15:09.619421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.619404    5419 scope.go:122] "RemoveContainer" containerID="8418048c7def218f7cfcca148e2f62afe8bb716b0d09462fb380c885666f74f5"
Oct 31 01:15:09.619683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.619660    5419 scope.go:122] "RemoveContainer" containerID="41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac"
Oct 31 01:15:09.619971 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:09.619945    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-tml7w_odh-ai-gateway-infra(512fe42e-1236-4f12-9e3a-f7cec4cc8eab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab"
Oct 31 01:15:09.621078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.621049    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj" event={"ID":"afaf2744-8709-4540-962b-a5de808bc38a","Type":"ContainerStarted","Data":"6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9"}
Oct 31 01:15:09.621210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.621109    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj" event={"ID":"afaf2744-8709-4540-962b-a5de808bc38a","Type":"ContainerStarted","Data":"7ce6fe8f4a00c286a5444ade84276e06e8dd6486221e921449c32322dcc1da57"}
Oct 31 01:15:09.621210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.621166    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:09.623007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.622991    5419 scope.go:122] "RemoveContainer" containerID="3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a"
Oct 31 01:15:09.623217 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:09.623200    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb_odh-ai-gateway-infra(63a9919d-851e-4eca-bd12-6b1d6496ad63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63"
Oct 31 01:15:09.736388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.736335    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf" podStartSLOduration=1.73631927 podStartE2EDuration="1.73631927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:09.733997244 +0000 UTC m=+631.034953817" watchObservedRunningTime="2026-10-31 01:15:09.73631927 +0000 UTC m=+631.037275848"
Oct 31 01:15:09.968732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:09.968637    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj" podStartSLOduration=1.968622824 podStartE2EDuration="1.968622824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:09.967509991 +0000 UTC m=+631.268466569" watchObservedRunningTime="2026-10-31 01:15:09.968622824 +0000 UTC m=+631.269579400"
Oct 31 01:15:10.630013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:10.629978    5419 scope.go:122] "RemoveContainer" containerID="41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac"
Oct 31 01:15:10.630442 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:10.630288    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-tml7w_odh-ai-gateway-infra(512fe42e-1236-4f12-9e3a-f7cec4cc8eab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab"
Oct 31 01:15:12.646622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.646585    5419 generic.go:356] "Generic (PLEG): container finished" podID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerID="d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826" exitCode=0
Oct 31 01:15:12.647071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.646657    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" event={"ID":"00630b5c-589c-4b00-b8c2-e4055c355c9f","Type":"ContainerDied","Data":"d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"}
Oct 31 01:15:12.647071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.647024    5419 scope.go:122] "RemoveContainer" containerID="d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"
Oct 31 01:15:12.648269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.648245    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:12.648329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.648308    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:12.648363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.648339    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:12.749870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:12.749837    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:15:13.405374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:13.405335    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:15:13.655589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:13.655509    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" event={"ID":"00630b5c-589c-4b00-b8c2-e4055c355c9f","Type":"ContainerStarted","Data":"cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3"}
Oct 31 01:15:14.110529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:14.110496    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:15:14.119577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:14.119543    5419 patch_prober.go:29] interesting pod/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:15:14.119913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:14.119862    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:15:14.633242 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:14.633063    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:15:15.128160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:15.126560    5419 patch_prober.go:29] interesting pod/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:15:15.128160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:15.126625    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:15:15.555193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:15.555133    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:15:16.110162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:16.109701    5419 patch_prober.go:29] interesting pod/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:15:16.121158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:16.118518    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:15:17.118307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:17.109560    5419 patch_prober.go:29] interesting pod/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:15:17.118307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:17.109625    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:15:18.112578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:18.112538    5419 patch_prober.go:29] interesting pod/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 01:15:18.112875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:18.112847    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 01:15:19.113762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:19.113730    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:15:19.114227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:19.114169    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:15:19.114953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:19.114937    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:15:20.463266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:20.463232    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:15:20.633799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:20.633760    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:15:20.634209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:20.634182    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:15:20.830639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:20.830607    5419 scope.go:122] "RemoveContainer" containerID="41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac"
Oct 31 01:15:21.724464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:21.724428    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerStarted","Data":"a246f7b2e21c991d813b1fbc10ab09251ee8c2b9ff56b3782d3983376bb26cb9"}
Oct 31 01:15:21.830594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:21.830563    5419 scope.go:122] "RemoveContainer" containerID="3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a"
Oct 31 01:15:22.730961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:22.730923    5419 generic.go:356] "Generic (PLEG): container finished" podID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerID="a246f7b2e21c991d813b1fbc10ab09251ee8c2b9ff56b3782d3983376bb26cb9" exitCode=60
Oct 31 01:15:22.733572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:22.730997    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerDied","Data":"a246f7b2e21c991d813b1fbc10ab09251ee8c2b9ff56b3782d3983376bb26cb9"}
Oct 31 01:15:22.733572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:22.731062    5419 scope.go:122] "RemoveContainer" containerID="41346ca195508df666c53562406f8bbc9d1989784d182062c362b83bd5e56cac"
Oct 31 01:15:22.733572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:22.731377    5419 scope.go:122] "RemoveContainer" containerID="a246f7b2e21c991d813b1fbc10ab09251ee8c2b9ff56b3782d3983376bb26cb9"
Oct 31 01:15:22.733572 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:22.731607    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890155-tml7w_odh-ai-gateway-infra(512fe42e-1236-4f12-9e3a-f7cec4cc8eab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab"
Oct 31 01:15:22.733572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:22.732996    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerStarted","Data":"c4896f7f515e0a76f1c4c9485b14ed18811a72886a853efd11d432f33ad7776e"}
Oct 31 01:15:22.858056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:22.857981    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"]
Oct 31 01:15:23.739208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.739178    5419 generic.go:356] "Generic (PLEG): container finished" podID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerID="9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466" exitCode=0
Oct 31 01:15:23.739714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.739253    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" event={"ID":"63ca9a39-e693-418a-b9ed-5f64ac4475b4","Type":"ContainerDied","Data":"9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"}
Oct 31 01:15:23.739714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.739643    5419 scope.go:122] "RemoveContainer" containerID="9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"
Oct 31 01:15:23.740923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.740897    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:23.741027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.740983    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:23.741090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.741030    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:23.741180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.741131    5419 generic.go:356] "Generic (PLEG): container finished" podID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerID="c4896f7f515e0a76f1c4c9485b14ed18811a72886a853efd11d432f33ad7776e" exitCode=60
Oct 31 01:15:23.741253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.741162    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerDied","Data":"c4896f7f515e0a76f1c4c9485b14ed18811a72886a853efd11d432f33ad7776e"}
Oct 31 01:15:23.741310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.741286    5419 scope.go:122] "RemoveContainer" containerID="3a81a776b912b611f0811477c8dadb834ca74b6e9c6bab4e63f08d79f9460c9a"
Oct 31 01:15:23.741627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.741518    5419 scope.go:122] "RemoveContainer" containerID="c4896f7f515e0a76f1c4c9485b14ed18811a72886a853efd11d432f33ad7776e"
Oct 31 01:15:23.741725 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:23.741706    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb_odh-ai-gateway-infra(63a9919d-851e-4eca-bd12-6b1d6496ad63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63"
Oct 31 01:15:23.883588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.883562    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:23.928903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.928280    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm\" (UniqueName: \"kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm\") pod \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") "
Oct 31 01:15:23.928903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.928353    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca\") pod \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\" (UID: \"512fe42e-1236-4f12-9e3a-f7cec4cc8eab\") "
Oct 31 01:15:23.929339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.929313    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca" pod "512fe42e-1236-4f12-9e3a-f7cec4cc8eab" (UID: "512fe42e-1236-4f12-9e3a-f7cec4cc8eab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:23.931464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.931435    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm" pod "512fe42e-1236-4f12-9e3a-f7cec4cc8eab" (UID: "512fe42e-1236-4f12-9e3a-f7cec4cc8eab"). InnerVolumeSpecName "kube-api-access-65dkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:23.967573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:23.967538    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"]
Oct 31 01:15:24.029268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.029199    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65dkm\" (UniqueName: \"kubernetes.io/projected/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-kube-api-access-65dkm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:24.029268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.029226    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512fe42e-1236-4f12-9e3a-f7cec4cc8eab-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:24.751120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.751079    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" event={"ID":"63ca9a39-e693-418a-b9ed-5f64ac4475b4","Type":"ContainerStarted","Data":"54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f"}
Oct 31 01:15:24.763402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.763368    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w" event={"ID":"512fe42e-1236-4f12-9e3a-f7cec4cc8eab","Type":"ContainerDied","Data":"6a9b545fd33ec6abff80251f4ec85eb31107958f13bd9ab93d25fb907847da44"}
Oct 31 01:15:24.763542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.763432    5419 scope.go:122] "RemoveContainer" containerID="a246f7b2e21c991d813b1fbc10ab09251ee8c2b9ff56b3782d3983376bb26cb9"
Oct 31 01:15:24.763542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.763390    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"
Oct 31 01:15:24.957009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:24.956984    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:25.000274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.000231    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:15:25.000476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.000348    5419 patch_prober.go:29] interesting pod/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:15:25.000476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.000389    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:15:25.000683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.000651    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"]
Oct 31 01:15:25.002103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.002051    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890155-tml7w"]
Oct 31 01:15:25.038880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.038836    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca\") pod \"63a9919d-851e-4eca-bd12-6b1d6496ad63\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") "
Oct 31 01:15:25.039068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.038966    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp\" (UniqueName: \"kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp\") pod \"63a9919d-851e-4eca-bd12-6b1d6496ad63\" (UID: \"63a9919d-851e-4eca-bd12-6b1d6496ad63\") "
Oct 31 01:15:25.039210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.039189    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca" pod "63a9919d-851e-4eca-bd12-6b1d6496ad63" (UID: "63a9919d-851e-4eca-bd12-6b1d6496ad63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:25.039288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.039274    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a9919d-851e-4eca-bd12-6b1d6496ad63-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:25.041232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.041209    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp" pod "63a9919d-851e-4eca-bd12-6b1d6496ad63" (UID: "63a9919d-851e-4eca-bd12-6b1d6496ad63"). InnerVolumeSpecName "kube-api-access-cpjgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:25.140505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.140472    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjgp\" (UniqueName: \"kubernetes.io/projected/63a9919d-851e-4eca-bd12-6b1d6496ad63-kube-api-access-cpjgp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:25.770371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.770342    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"
Oct 31 01:15:25.770871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.770374    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb" event={"ID":"63a9919d-851e-4eca-bd12-6b1d6496ad63","Type":"ContainerDied","Data":"02703394d63d55361943a57a8de1730228cc98dd7c8027507edfedbe45e4d38a"}
Oct 31 01:15:25.770871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.770434    5419 scope.go:122] "RemoveContainer" containerID="c4896f7f515e0a76f1c4c9485b14ed18811a72886a853efd11d432f33ad7776e"
Oct 31 01:15:25.954252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.954209    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"]
Oct 31 01:15:25.958588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.958564    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-a0c7a371-29890155-txpdb"]
Oct 31 01:15:25.999296 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.999262    5419 patch_prober.go:29] interesting pod/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:15:25.999479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:25.999318    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:15:26.287164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.286814    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"]
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287464    5419 cpu_manager.go:414] "containerMap: removing container" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287487    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287517    5419 cpu_manager.go:414] "containerMap: removing container" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287525    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287551    5419 cpu_manager.go:414] "containerMap: removing container" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287560    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287570    5419 cpu_manager.go:414] "containerMap: removing container" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287577    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287685    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287707    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287720    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287733    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.288167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.287741    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:26.303456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.303431    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.346193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.346170    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"]
Oct 31 01:15:26.350950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.350926    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.351087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.351003    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.351166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.351100    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf27\" (UniqueName: \"kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.355654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.355632    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:15:26.355771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.355669    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:15:26.356119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.356097    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:15:26.356235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.356149    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:15:26.451897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.451868    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.452110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.451925    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf27\" (UniqueName: \"kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.452110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.451959    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.452588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.452563    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.454878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.454855    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.509306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.509276    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf27\" (UniqueName: \"kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27\") pod \"maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.615635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.615537    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:26.836691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.836661    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" path="/var/lib/kubelet/pods/512fe42e-1236-4f12-9e3a-f7cec4cc8eab/volumes"
Oct 31 01:15:26.837037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.837017    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" path="/var/lib/kubelet/pods/63a9919d-851e-4eca-bd12-6b1d6496ad63/volumes"
Oct 31 01:15:26.993503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.993461    5419 patch_prober.go:29] interesting pod/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:15:26.993706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:26.993525    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:15:27.004372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:27.002455    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"]
Oct 31 01:15:27.809179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:27.809130    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh" event={"ID":"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc","Type":"ContainerStarted","Data":"8156c152a86d049b3ef604912bfa351ad8f1464e15595bcc3c5ae5cfe3ed3f1d"}
Oct 31 01:15:27.809345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:27.809203    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh" event={"ID":"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc","Type":"ContainerStarted","Data":"f4c9a7cb5593e22426da609a6bb047f81d88ca2d27b310e156729490db04601a"}
Oct 31 01:15:27.922722 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:27.922669    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh" podStartSLOduration=1.922653543 podStartE2EDuration="1.922653543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:27.921667433 +0000 UTC m=+649.222624011" watchObservedRunningTime="2026-10-31 01:15:27.922653543 +0000 UTC m=+649.223610121"
Oct 31 01:15:27.994162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:27.993508    5419 patch_prober.go:29] interesting pod/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:15:27.994162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:27.993568    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:15:28.817248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:28.817207    5419 generic.go:356] "Generic (PLEG): container finished" podID="eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" containerID="8156c152a86d049b3ef604912bfa351ad8f1464e15595bcc3c5ae5cfe3ed3f1d" exitCode=0
Oct 31 01:15:28.817437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:28.817298    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh" event={"ID":"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc","Type":"ContainerDied","Data":"8156c152a86d049b3ef604912bfa351ad8f1464e15595bcc3c5ae5cfe3ed3f1d"}
Oct 31 01:15:28.982524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:28.982488    5419 patch_prober.go:29] interesting pod/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 01:15:28.983098 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:28.983068    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 01:15:30.001829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.000426    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:15:30.001829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.001767    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:15:30.006844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.006809    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:15:30.032129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.032102    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:30.116415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.116326    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27\" (UniqueName: \"kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27\") pod \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") "
Oct 31 01:15:30.116415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.116377    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca\") pod \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") "
Oct 31 01:15:30.116629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.116482    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca\") pod \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\" (UID: \"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc\") "
Oct 31 01:15:30.116719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.116695    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca" pod "eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" (UID: "eb3279c5-2194-4d11-bff2-b69e4b7aa2fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:30.116913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.116890    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:30.118752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.118731    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca" pod "eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" (UID: "eb3279c5-2194-4d11-bff2-b69e4b7aa2fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:30.118840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.118750    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27" pod "eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" (UID: "eb3279c5-2194-4d11-bff2-b69e4b7aa2fc"). InnerVolumeSpecName "kube-api-access-mjf27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:30.217952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.217917    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:30.217952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.217949    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjf27\" (UniqueName: \"kubernetes.io/projected/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc-kube-api-access-mjf27\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:30.829937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.829905    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"
Oct 31 01:15:30.837084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.837049    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh" event={"ID":"eb3279c5-2194-4d11-bff2-b69e4b7aa2fc","Type":"ContainerDied","Data":"f4c9a7cb5593e22426da609a6bb047f81d88ca2d27b310e156729490db04601a"}
Oct 31 01:15:30.837278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:30.837101    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c9a7cb5593e22426da609a6bb047f81d88ca2d27b310e156729490db04601a"
Oct 31 01:15:36.600412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.600373    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"]
Oct 31 01:15:36.601050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601029    5419 cpu_manager.go:414] "containerMap: removing container" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:36.601095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601055    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:36.601095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601081    5419 cpu_manager.go:414] "containerMap: removing container" podUID="eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" containerName="revoke-keys"
Oct 31 01:15:36.601095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601089    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" containerName="revoke-keys"
Oct 31 01:15:36.601218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601104    5419 cpu_manager.go:414] "containerMap: removing container" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:36.601218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601112    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a9919d-851e-4eca-bd12-6b1d6496ad63" containerName="cleanup"
Oct 31 01:15:36.601278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601269    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="512fe42e-1236-4f12-9e3a-f7cec4cc8eab" containerName="cleanup"
Oct 31 01:15:36.601311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.601286    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" containerName="revoke-keys"
Oct 31 01:15:36.604997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.604976    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.660359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.660322    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"]
Oct 31 01:15:36.677106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.677076    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.677390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.677172    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqwn\" (UniqueName: \"kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.677390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.677311    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.707178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.707151    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:15:36.707321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.707132    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:15:36.707321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.707155    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:15:36.707321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.707159    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:15:36.778221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.778184    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqwn\" (UniqueName: \"kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.778385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.778285    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.778385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.778339    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.779117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.779088    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.780969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.780938    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.835681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.835641    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqwn\" (UniqueName: \"kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn\") pod \"maas-api-revoke-keys-e2e-ait-f94871a1-79xh8\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:36.918266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:36.918177    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:37.295794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.295683    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"]
Oct 31 01:15:37.297119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.297093    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:15:37.298731 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:37.298592    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322704de_c256_47ff_b401_6e193777abac.slice/crio-a633466a81a7b82e1bd1a871a5fabab145d15f6d7ef8a6e4cd9459d3194a9c54 WatchSource:0}: Error finding container a633466a81a7b82e1bd1a871a5fabab145d15f6d7ef8a6e4cd9459d3194a9c54: Status 404 returned error can't find the container with id a633466a81a7b82e1bd1a871a5fabab145d15f6d7ef8a6e4cd9459d3194a9c54
Oct 31 01:15:37.438720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.438349    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"]
Oct 31 01:15:37.866463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.866432    5419 generic.go:356] "Generic (PLEG): container finished" podID="26586095-56c1-4668-a769-0b96f2aecf4f" containerID="ae3a066f7246a45efc970e4a7104cb054627800a214933a43ad4b69f187aa2d0" exitCode=0
Oct 31 01:15:37.866881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.866503    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" event={"ID":"26586095-56c1-4668-a769-0b96f2aecf4f","Type":"ContainerDied","Data":"ae3a066f7246a45efc970e4a7104cb054627800a214933a43ad4b69f187aa2d0"}
Oct 31 01:15:37.866961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.866946    5419 scope.go:122] "RemoveContainer" containerID="ae3a066f7246a45efc970e4a7104cb054627800a214933a43ad4b69f187aa2d0"
Oct 31 01:15:37.868385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.868355    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:37.868500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.868391    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8" event={"ID":"322704de-c256-47ff-b401-6e193777abac","Type":"ContainerStarted","Data":"5a647bddbd5ca6797f89c3c545a23f9896de2645020484696806cf0196169742"}
Oct 31 01:15:37.868500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.868431    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:37.868500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.868438    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8" event={"ID":"322704de-c256-47ff-b401-6e193777abac","Type":"ContainerStarted","Data":"a633466a81a7b82e1bd1a871a5fabab145d15f6d7ef8a6e4cd9459d3194a9c54"}
Oct 31 01:15:37.868500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:37.868464    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:38.097450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:38.097317    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8" podStartSLOduration=2.09730068 podStartE2EDuration="2.09730068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:38.096777573 +0000 UTC m=+659.397734154" watchObservedRunningTime="2026-10-31 01:15:38.09730068 +0000 UTC m=+659.398257257"
Oct 31 01:15:38.470594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:38.470572    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:15:38.904046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:38.903939    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" event={"ID":"26586095-56c1-4668-a769-0b96f2aecf4f","Type":"ContainerStarted","Data":"44780bafa93f934b0e3e51356af89d7580dae7357bfe8d5b32b17dc81ebc22c4"}
Oct 31 01:15:38.905753 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:38.905725    5419 generic.go:356] "Generic (PLEG): container finished" podID="322704de-c256-47ff-b401-6e193777abac" containerID="5a647bddbd5ca6797f89c3c545a23f9896de2645020484696806cf0196169742" exitCode=0
Oct 31 01:15:38.905903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:38.905809    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8" event={"ID":"322704de-c256-47ff-b401-6e193777abac","Type":"ContainerDied","Data":"5a647bddbd5ca6797f89c3c545a23f9896de2645020484696806cf0196169742"}
Oct 31 01:15:39.299261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:39.299211    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:15:39.299523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:39.299500    5419 patch_prober.go:29] interesting pod/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:15:39.299577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:39.299552    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:15:40.062508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.062484    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:40.121546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.121502    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn\" (UniqueName: \"kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn\") pod \"322704de-c256-47ff-b401-6e193777abac\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") "
Oct 31 01:15:40.121546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.121552    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca\") pod \"322704de-c256-47ff-b401-6e193777abac\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") "
Oct 31 01:15:40.121759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.121580    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca\") pod \"322704de-c256-47ff-b401-6e193777abac\" (UID: \"322704de-c256-47ff-b401-6e193777abac\") "
Oct 31 01:15:40.121987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.121958    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca" pod "322704de-c256-47ff-b401-6e193777abac" (UID: "322704de-c256-47ff-b401-6e193777abac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:40.123780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.123750    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca" pod "322704de-c256-47ff-b401-6e193777abac" (UID: "322704de-c256-47ff-b401-6e193777abac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:40.123874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.123800    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn" pod "322704de-c256-47ff-b401-6e193777abac" (UID: "322704de-c256-47ff-b401-6e193777abac"). InnerVolumeSpecName "kube-api-access-6gqwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:40.223060 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.223024    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqwn\" (UniqueName: \"kubernetes.io/projected/322704de-c256-47ff-b401-6e193777abac-kube-api-access-6gqwn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:40.223060 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.223065    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/322704de-c256-47ff-b401-6e193777abac-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:40.223267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.223076    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/322704de-c256-47ff-b401-6e193777abac-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:40.299166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.298182    5419 patch_prober.go:29] interesting pod/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:15:40.299166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.298245    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:15:40.916837 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.916799    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8" event={"ID":"322704de-c256-47ff-b401-6e193777abac","Type":"ContainerDied","Data":"a633466a81a7b82e1bd1a871a5fabab145d15f6d7ef8a6e4cd9459d3194a9c54"}
Oct 31 01:15:40.917018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.916858    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a633466a81a7b82e1bd1a871a5fabab145d15f6d7ef8a6e4cd9459d3194a9c54"
Oct 31 01:15:40.917018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:40.916822    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"
Oct 31 01:15:41.299167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:41.297601    5419 patch_prober.go:29] interesting pod/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:15:41.299167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:41.297679    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:15:42.296164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.295492    5419 patch_prober.go:29] interesting pod/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:15:42.296164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.295553    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:15:42.439670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.439608    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" podUID="d232b064-7f45-4696-9eab-5547972d6c1c" containerName="payload-pre-processing" containerID="cri-o://0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc" gracePeriod=25
Oct 31 01:15:42.466780 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:42.466764    5419 logging.go:55] [core] [Channel #158 SubChannel #159]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.37:50052", ServerName: "10.133.0.37:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.37:50052: connect: connection refused"
Oct 31 01:15:42.747463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.747438    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:15:42.847702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.847614    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume\") pod \"d232b064-7f45-4696-9eab-5547972d6c1c\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") "
Oct 31 01:15:42.847702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.847648    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm\" (UniqueName: \"kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm\") pod \"d232b064-7f45-4696-9eab-5547972d6c1c\" (UID: \"d232b064-7f45-4696-9eab-5547972d6c1c\") "
Oct 31 01:15:42.848010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.847986    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume" pod "d232b064-7f45-4696-9eab-5547972d6c1c" (UID: "d232b064-7f45-4696-9eab-5547972d6c1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:42.849950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.849922    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm" pod "d232b064-7f45-4696-9eab-5547972d6c1c" (UID: "d232b064-7f45-4696-9eab-5547972d6c1c"). InnerVolumeSpecName "kube-api-access-gccpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:42.930989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.930958    5419 generic.go:356] "Generic (PLEG): container finished" podID="d232b064-7f45-4696-9eab-5547972d6c1c" containerID="0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc" exitCode=0
Oct 31 01:15:42.931173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.931005    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"
Oct 31 01:15:42.931173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.931049    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" event={"ID":"d232b064-7f45-4696-9eab-5547972d6c1c","Type":"ContainerDied","Data":"0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc"}
Oct 31 01:15:42.931173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.931101    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" event={"ID":"d232b064-7f45-4696-9eab-5547972d6c1c","Type":"ContainerDied","Data":"9a2fb1d324184258f3de2e27e44f0ebd76ccca2b68c46d55b0b3aa3687e2ac48"}
Oct 31 01:15:42.931173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.931123    5419 scope.go:122] "RemoveContainer" containerID="0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc"
Oct 31 01:15:42.940028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.940008    5419 scope.go:122] "RemoveContainer" containerID="0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc"
Oct 31 01:15:42.940309 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:42.940292    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc\": container with ID starting with 0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc not found: ID does not exist" containerID="0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc"
Oct 31 01:15:42.940363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.940317    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc"} err="failed to get container status \"0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc\": rpc error: code = NotFound desc = could not find container \"0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc\": container with ID starting with 0061840ca4d8e9ab8c3546dc33ae2f0a77dad820192fb494d595f77aa2c0eedc not found: ID does not exist"
Oct 31 01:15:42.948793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.948775    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d232b064-7f45-4696-9eab-5547972d6c1c-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:42.948842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:42.948796    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gccpm\" (UniqueName: \"kubernetes.io/projected/d232b064-7f45-4696-9eab-5547972d6c1c-kube-api-access-gccpm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:43.101637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:43.101560    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"]
Oct 31 01:15:43.106178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:43.106122    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l"]
Oct 31 01:15:43.295983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:43.295512    5419 patch_prober.go:29] interesting pod/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 01:15:43.295983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:43.295587    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 01:15:43.467478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:43.467388    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a0c7a371-7b6d5489b7-8774l" podUID="d232b064-7f45-4696-9eab-5547972d6c1c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.37:50052\" within 1s: context deadline exceeded"
Oct 31 01:15:44.297095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:44.297065    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:15:44.297387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:44.297365    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:15:44.298018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:44.298000    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:15:44.581034 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:44.580953    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:15:44.836168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:44.836076    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d232b064-7f45-4696-9eab-5547972d6c1c" path="/var/lib/kubelet/pods/d232b064-7f45-4696-9eab-5547972d6c1c/volumes"
Oct 31 01:15:49.637865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:49.637833    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"]
Oct 31 01:15:49.687800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:49.687767    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a0c7a371-q5zfh"]
Oct 31 01:15:50.074724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.074688    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"]
Oct 31 01:15:50.075086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.075073    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d232b064-7f45-4696-9eab-5547972d6c1c" containerName="payload-pre-processing"
Oct 31 01:15:50.075129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.075087    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d232b064-7f45-4696-9eab-5547972d6c1c" containerName="payload-pre-processing"
Oct 31 01:15:50.075129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.075106    5419 cpu_manager.go:414] "containerMap: removing container" podUID="322704de-c256-47ff-b401-6e193777abac" containerName="revoke-keys"
Oct 31 01:15:50.075129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.075111    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322704de-c256-47ff-b401-6e193777abac" containerName="revoke-keys"
Oct 31 01:15:50.075238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.075197    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d232b064-7f45-4696-9eab-5547972d6c1c" containerName="payload-pre-processing"
Oct 31 01:15:50.075238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.075206    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="322704de-c256-47ff-b401-6e193777abac" containerName="revoke-keys"
Oct 31 01:15:50.079522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.079504    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.133555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.133525    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:15:50.182728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.182695    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:15:50.182900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.182737    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:15:50.182900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.182758    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:15:50.184460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.184442    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"]
Oct 31 01:15:50.218760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.218733    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.218885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.218785    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6tf\" (UniqueName: \"kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.218938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.218920    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.319938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.319883    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.320121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.319969    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.320202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.320125    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6tf\" (UniqueName: \"kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.320647 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.320629    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.322490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.322473    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.377023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.376948    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6tf\" (UniqueName: \"kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf\") pod \"maas-api-revoke-keys-e2e-ait-7164dd00-79k2g\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.391818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.391791    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:50.759730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.759681    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"]
Oct 31 01:15:50.762221 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:50.762192    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd873d741_d410_4f63_8c5c_c7b62d8d3b4e.slice/crio-f808707747c277e267d9940b575818a920fa6c31afb7906b6d989964612a5df5 WatchSource:0}: Error finding container f808707747c277e267d9940b575818a920fa6c31afb7906b6d989964612a5df5: Status 404 returned error can't find the container with id f808707747c277e267d9940b575818a920fa6c31afb7906b6d989964612a5df5
Oct 31 01:15:50.835900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.835867    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3279c5-2194-4d11-bff2-b69e4b7aa2fc" path="/var/lib/kubelet/pods/eb3279c5-2194-4d11-bff2-b69e4b7aa2fc/volumes"
Oct 31 01:15:50.973882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:50.973845    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g" event={"ID":"d873d741-d410-4f63-8c5c-c7b62d8d3b4e","Type":"ContainerStarted","Data":"f808707747c277e267d9940b575818a920fa6c31afb7906b6d989964612a5df5"}
Oct 31 01:15:51.084520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.084500    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"]
Oct 31 01:15:51.087854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.087838    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.139993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.139455    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a8e157-openshift-default-dockercfg-f24hn\""
Oct 31 01:15:51.188509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.188479    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"]
Oct 31 01:15:51.234157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234098    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234162    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234198    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234312    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234371    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234444    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234474    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234495    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.234659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.234510    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6d66\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336273    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336345    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336381    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336409    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6d66\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336464    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336498    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336554    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336597    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.336648    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.337778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.337367    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.338455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.338432    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.338775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.338748    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.338872 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.338816    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.339031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.339015    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.340310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.340289    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.341165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.341114    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.394414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.394379    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.394994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.394974    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6d66\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.400794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.400778    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:51.605711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.605627    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"]
Oct 31 01:15:51.722095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.722066    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"]
Oct 31 01:15:51.822202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.822172    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"]
Oct 31 01:15:51.824569 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:51.824541    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5493b4_7690_4d3a_8b69_4f8105e82abc.slice/crio-7460b89ccf3b70f4c3f530e4fff591dcd19f407f27d82e7344df63284c9b459a WatchSource:0}: Error finding container 7460b89ccf3b70f4c3f530e4fff591dcd19f407f27d82e7344df63284c9b459a: Status 404 returned error can't find the container with id 7460b89ccf3b70f4c3f530e4fff591dcd19f407f27d82e7344df63284c9b459a
Oct 31 01:15:51.826995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.826960    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:51.827064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.827019    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:51.827064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.827050    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:15:51.980771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.980733    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" event={"ID":"ad5493b4-7690-4d3a-8b69-4f8105e82abc","Type":"ContainerStarted","Data":"8b8c5a2a157745558a1b41f3cab5f9c4a1033262e5655c84f418743c7643110e"}
Oct 31 01:15:51.981069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.981039    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" event={"ID":"ad5493b4-7690-4d3a-8b69-4f8105e82abc","Type":"ContainerStarted","Data":"7460b89ccf3b70f4c3f530e4fff591dcd19f407f27d82e7344df63284c9b459a"}
Oct 31 01:15:51.982585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.982552    5419 generic.go:356] "Generic (PLEG): container finished" podID="d873d741-d410-4f63-8c5c-c7b62d8d3b4e" containerID="d884f8dfedef90c84b323e90412d181d50b51f1dcf4501cfb9e92a364c5ba9b7" exitCode=0
Oct 31 01:15:51.982698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:51.982618    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g" event={"ID":"d873d741-d410-4f63-8c5c-c7b62d8d3b4e","Type":"ContainerDied","Data":"d884f8dfedef90c84b323e90412d181d50b51f1dcf4501cfb9e92a364c5ba9b7"}
Oct 31 01:15:52.100874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.100821    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podStartSLOduration=1.100808629 podStartE2EDuration="1.100808629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:52.09830526 +0000 UTC m=+673.399261836" watchObservedRunningTime="2026-10-31 01:15:52.100808629 +0000 UTC m=+673.401765560"
Oct 31 01:15:52.401076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.401038    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:52.406181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.406158    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:52.511987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.511955    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"]
Oct 31 01:15:52.515465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.515449    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.573007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.572975    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"]
Oct 31 01:15:52.649644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.649551    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.649644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.649594    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.649858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.649731    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gzgr\" (UniqueName: \"kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.750637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.750597    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.750831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.750655    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.750831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.750744    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gzgr\" (UniqueName: \"kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.751254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.751230    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.753149 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.753112    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.811173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.811117    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gzgr\" (UniqueName: \"kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr\") pod \"maas-api-revoke-keys-e2e-derive-b9656f65-fth8z\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.828298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.828273    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:52.989396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.989122    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:52.990829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:52.990809    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:15:53.142314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.142288    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:53.229862 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.229837    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"]
Oct 31 01:15:53.231479 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:15:53.231451    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0157f954_14b1_45f5_8042_09e7723d3c86.slice/crio-9121d8843d3375ac054a45c2a85b33364494094b1d920abf392a54f035c401d2 WatchSource:0}: Error finding container 9121d8843d3375ac054a45c2a85b33364494094b1d920abf392a54f035c401d2: Status 404 returned error can't find the container with id 9121d8843d3375ac054a45c2a85b33364494094b1d920abf392a54f035c401d2
Oct 31 01:15:53.260010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.259550    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf\" (UniqueName: \"kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf\") pod \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") "
Oct 31 01:15:53.260010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.259682    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca\") pod \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") "
Oct 31 01:15:53.260010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.259791    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca\") pod \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\" (UID: \"d873d741-d410-4f63-8c5c-c7b62d8d3b4e\") "
Oct 31 01:15:53.260799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.260769    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca" pod "d873d741-d410-4f63-8c5c-c7b62d8d3b4e" (UID: "d873d741-d410-4f63-8c5c-c7b62d8d3b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:53.262866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.262846    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca" pod "d873d741-d410-4f63-8c5c-c7b62d8d3b4e" (UID: "d873d741-d410-4f63-8c5c-c7b62d8d3b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:53.263058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.263016    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf" pod "d873d741-d410-4f63-8c5c-c7b62d8d3b4e" (UID: "d873d741-d410-4f63-8c5c-c7b62d8d3b4e"). InnerVolumeSpecName "kube-api-access-zh6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:53.360935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.360818    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:53.360935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.360867    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh6tf\" (UniqueName: \"kubernetes.io/projected/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-kube-api-access-zh6tf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:53.360935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.360883    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d873d741-d410-4f63-8c5c-c7b62d8d3b4e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:53.995838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.995805    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"
Oct 31 01:15:53.995838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.995826    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g" event={"ID":"d873d741-d410-4f63-8c5c-c7b62d8d3b4e","Type":"ContainerDied","Data":"f808707747c277e267d9940b575818a920fa6c31afb7906b6d989964612a5df5"}
Oct 31 01:15:53.996315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.995878    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f808707747c277e267d9940b575818a920fa6c31afb7906b6d989964612a5df5"
Oct 31 01:15:53.997412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.997385    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z" event={"ID":"0157f954-14b1-45f5-8042-09e7723d3c86","Type":"ContainerStarted","Data":"a329d9917bcaa5012f353150f4f08353e9b966f59a51a880f9770b576804627d"}
Oct 31 01:15:53.997529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:53.997436    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z" event={"ID":"0157f954-14b1-45f5-8042-09e7723d3c86","Type":"ContainerStarted","Data":"9121d8843d3375ac054a45c2a85b33364494094b1d920abf392a54f035c401d2"}
Oct 31 01:15:54.111455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:54.111405    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z" podStartSLOduration=2.111389822 podStartE2EDuration="2.111389822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:54.109636819 +0000 UTC m=+675.410593395" watchObservedRunningTime="2026-10-31 01:15:54.111389822 +0000 UTC m=+675.412346418"
Oct 31 01:15:55.004927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:55.004896    5419 generic.go:356] "Generic (PLEG): container finished" podID="0157f954-14b1-45f5-8042-09e7723d3c86" containerID="a329d9917bcaa5012f353150f4f08353e9b966f59a51a880f9770b576804627d" exitCode=0
Oct 31 01:15:55.005344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:55.004959    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z" event={"ID":"0157f954-14b1-45f5-8042-09e7723d3c86","Type":"ContainerDied","Data":"a329d9917bcaa5012f353150f4f08353e9b966f59a51a880f9770b576804627d"}
Oct 31 01:15:55.121295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:55.121258    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:15:56.142670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.142648    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:56.292885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.292807    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca\") pod \"0157f954-14b1-45f5-8042-09e7723d3c86\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") "
Oct 31 01:15:56.293034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.292901    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr\" (UniqueName: \"kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr\") pod \"0157f954-14b1-45f5-8042-09e7723d3c86\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") "
Oct 31 01:15:56.293034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.292996    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca\") pod \"0157f954-14b1-45f5-8042-09e7723d3c86\" (UID: \"0157f954-14b1-45f5-8042-09e7723d3c86\") "
Oct 31 01:15:56.293396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.293369    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca" pod "0157f954-14b1-45f5-8042-09e7723d3c86" (UID: "0157f954-14b1-45f5-8042-09e7723d3c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:56.295116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.295093    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca" pod "0157f954-14b1-45f5-8042-09e7723d3c86" (UID: "0157f954-14b1-45f5-8042-09e7723d3c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:56.295186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.295169    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr" pod "0157f954-14b1-45f5-8042-09e7723d3c86" (UID: "0157f954-14b1-45f5-8042-09e7723d3c86"). InnerVolumeSpecName "kube-api-access-9gzgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:56.394599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.394566    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:56.394599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.394593    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0157f954-14b1-45f5-8042-09e7723d3c86-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:56.394599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.394604    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gzgr\" (UniqueName: \"kubernetes.io/projected/0157f954-14b1-45f5-8042-09e7723d3c86-kube-api-access-9gzgr\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:56.605988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.605906    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q" podUID="47700bc2-37d4-4dd0-8298-fb7ad293a978" containerName="payload-processing" containerID="cri-o://5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672" gracePeriod=25
Oct 31 01:15:56.723006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.722963    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg" podUID="7382b896-814e-4467-9813-f78e881d0f85" containerName="payload-pre-processing" containerID="cri-o://11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b" gracePeriod=25
Oct 31 01:15:56.844326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.844304    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:15:56.955124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.955101    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:15:56.999585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.999550    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2\" (UniqueName: \"kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2\") pod \"47700bc2-37d4-4dd0-8298-fb7ad293a978\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") "
Oct 31 01:15:56.999811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:56.999783    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume\") pod \"47700bc2-37d4-4dd0-8298-fb7ad293a978\" (UID: \"47700bc2-37d4-4dd0-8298-fb7ad293a978\") "
Oct 31 01:15:57.000166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.000133    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume" pod "47700bc2-37d4-4dd0-8298-fb7ad293a978" (UID: "47700bc2-37d4-4dd0-8298-fb7ad293a978"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:57.001767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.001745    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2" pod "47700bc2-37d4-4dd0-8298-fb7ad293a978" (UID: "47700bc2-37d4-4dd0-8298-fb7ad293a978"). InnerVolumeSpecName "kube-api-access-rhvj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:57.016678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.016650    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"
Oct 31 01:15:57.016808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.016654    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z" event={"ID":"0157f954-14b1-45f5-8042-09e7723d3c86","Type":"ContainerDied","Data":"9121d8843d3375ac054a45c2a85b33364494094b1d920abf392a54f035c401d2"}
Oct 31 01:15:57.016808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.016786    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9121d8843d3375ac054a45c2a85b33364494094b1d920abf392a54f035c401d2"
Oct 31 01:15:57.018054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.018031    5419 generic.go:356] "Generic (PLEG): container finished" podID="7382b896-814e-4467-9813-f78e881d0f85" containerID="11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b" exitCode=0
Oct 31 01:15:57.018167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.018080    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"
Oct 31 01:15:57.018227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.018129    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg" event={"ID":"7382b896-814e-4467-9813-f78e881d0f85","Type":"ContainerDied","Data":"11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b"}
Oct 31 01:15:57.018299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.018208    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg" event={"ID":"7382b896-814e-4467-9813-f78e881d0f85","Type":"ContainerDied","Data":"014b6443ef7167a80ae331856c025f341036337dddb26bb5b03ec7b2fdb72906"}
Oct 31 01:15:57.018299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.018260    5419 scope.go:122] "RemoveContainer" containerID="11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b"
Oct 31 01:15:57.019983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.019960    5419 generic.go:356] "Generic (PLEG): container finished" podID="47700bc2-37d4-4dd0-8298-fb7ad293a978" containerID="5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672" exitCode=0
Oct 31 01:15:57.020067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.019994    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q" event={"ID":"47700bc2-37d4-4dd0-8298-fb7ad293a978","Type":"ContainerDied","Data":"5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672"}
Oct 31 01:15:57.020067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.020029    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q" event={"ID":"47700bc2-37d4-4dd0-8298-fb7ad293a978","Type":"ContainerDied","Data":"5c7108e0acac639f473848ffc8fa59bc30366b5b3140b1636de733f24d0588d2"}
Oct 31 01:15:57.020192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.020089    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"
Oct 31 01:15:57.027415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.027386    5419 scope.go:122] "RemoveContainer" containerID="11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b"
Oct 31 01:15:57.027773 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:57.027753    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b\": container with ID starting with 11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b not found: ID does not exist" containerID="11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b"
Oct 31 01:15:57.027855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.027780    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b"} err="failed to get container status \"11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b\": rpc error: code = NotFound desc = could not find container \"11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b\": container with ID starting with 11a88783e68b035e50006b57658f7f06e888f4be2eb690c7fb81d43194b0646b not found: ID does not exist"
Oct 31 01:15:57.027855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.027799    5419 scope.go:122] "RemoveContainer" containerID="5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672"
Oct 31 01:15:57.035763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.035745    5419 scope.go:122] "RemoveContainer" containerID="5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672"
Oct 31 01:15:57.036025 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:15:57.035997    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672\": container with ID starting with 5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672 not found: ID does not exist" containerID="5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672"
Oct 31 01:15:57.036087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.036034    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672"} err="failed to get container status \"5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672\": rpc error: code = NotFound desc = could not find container \"5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672\": container with ID starting with 5328b8d59cd8f5a94419064140a29642e717d27be5447b2de9c2b37404e8b672 not found: ID does not exist"
Oct 31 01:15:57.100566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.100537    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g\" (UniqueName: \"kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g\") pod \"7382b896-814e-4467-9813-f78e881d0f85\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") "
Oct 31 01:15:57.100750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.100578    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume\") pod \"7382b896-814e-4467-9813-f78e881d0f85\" (UID: \"7382b896-814e-4467-9813-f78e881d0f85\") "
Oct 31 01:15:57.100869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.100849    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47700bc2-37d4-4dd0-8298-fb7ad293a978-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:57.100939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.100874    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvj2\" (UniqueName: \"kubernetes.io/projected/47700bc2-37d4-4dd0-8298-fb7ad293a978-kube-api-access-rhvj2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:57.101020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.100998    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume" pod "7382b896-814e-4467-9813-f78e881d0f85" (UID: "7382b896-814e-4467-9813-f78e881d0f85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:57.102919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.102895    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g" pod "7382b896-814e-4467-9813-f78e881d0f85" (UID: "7382b896-814e-4467-9813-f78e881d0f85"). InnerVolumeSpecName "kube-api-access-tts4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:57.184723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.184644    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"]
Oct 31 01:15:57.191378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.191352    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f94871a1-59f8f6cd77-c5g2q"]
Oct 31 01:15:57.202334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.202315    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7382b896-814e-4467-9813-f78e881d0f85-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:57.202420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.202337    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tts4g\" (UniqueName: \"kubernetes.io/projected/7382b896-814e-4467-9813-f78e881d0f85-kube-api-access-tts4g\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:57.437181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.437074    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"]
Oct 31 01:15:57.441560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:57.441536    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f94871a1-bf9ddb7c9-8svzg"]
Oct 31 01:15:58.835730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:58.835698    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47700bc2-37d4-4dd0-8298-fb7ad293a978" path="/var/lib/kubelet/pods/47700bc2-37d4-4dd0-8298-fb7ad293a978/volumes"
Oct 31 01:15:58.836113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:15:58.836033    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7382b896-814e-4467-9813-f78e881d0f85" path="/var/lib/kubelet/pods/7382b896-814e-4467-9813-f78e881d0f85/volumes"
Oct 31 01:16:02.212078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212044    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"]
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212442    5419 cpu_manager.go:414] "containerMap: removing container" podUID="47700bc2-37d4-4dd0-8298-fb7ad293a978" containerName="payload-processing"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212453    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47700bc2-37d4-4dd0-8298-fb7ad293a978" containerName="payload-processing"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212465    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d873d741-d410-4f63-8c5c-c7b62d8d3b4e" containerName="revoke-keys"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212471    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d873d741-d410-4f63-8c5c-c7b62d8d3b4e" containerName="revoke-keys"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212493    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0157f954-14b1-45f5-8042-09e7723d3c86" containerName="revoke-keys"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212499    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0157f954-14b1-45f5-8042-09e7723d3c86" containerName="revoke-keys"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212510    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7382b896-814e-4467-9813-f78e881d0f85" containerName="payload-pre-processing"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212515    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7382b896-814e-4467-9813-f78e881d0f85" containerName="payload-pre-processing"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212575    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="47700bc2-37d4-4dd0-8298-fb7ad293a978" containerName="payload-processing"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212584    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d873d741-d410-4f63-8c5c-c7b62d8d3b4e" containerName="revoke-keys"
Oct 31 01:16:02.212589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212593    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0157f954-14b1-45f5-8042-09e7723d3c86" containerName="revoke-keys"
Oct 31 01:16:02.212910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.212601    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7382b896-814e-4467-9813-f78e881d0f85" containerName="payload-pre-processing"
Oct 31 01:16:02.217245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.217226    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.254005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.253971    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48cb\" (UniqueName: \"kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.254181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.254036    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.254181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.254125    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.272196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.272167    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"]
Oct 31 01:16:02.318945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.318907    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:16:02.318945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.318923    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:16:02.318945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.318913    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:16:02.319177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.318973    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:16:02.355332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.355300    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.355470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.355369    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48cb\" (UniqueName: \"kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.355470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.355444    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.355936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.355914    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.357833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.357811    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.412881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.412851    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48cb\" (UniqueName: \"kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb\") pod \"maas-api-revoke-keys-e2e-ait-870eefeb-n77sg\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.514464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.514435    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"]
Oct 31 01:16:02.530185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.530131    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:02.902224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:02.902203    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"]
Oct 31 01:16:02.904453 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:02.904427    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adc9451_975f_4d36_986e_97a7f0542554.slice/crio-7dd488cd74acd845ccea581e797fc503aef50a04e17a4f82b62475e1ca1f9f96 WatchSource:0}: Error finding container 7dd488cd74acd845ccea581e797fc503aef50a04e17a4f82b62475e1ca1f9f96: Status 404 returned error can't find the container with id 7dd488cd74acd845ccea581e797fc503aef50a04e17a4f82b62475e1ca1f9f96
Oct 31 01:16:03.059583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:03.059543    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg" event={"ID":"8adc9451-975f-4d36-986e-97a7f0542554","Type":"ContainerStarted","Data":"7dd488cd74acd845ccea581e797fc503aef50a04e17a4f82b62475e1ca1f9f96"}
Oct 31 01:16:04.066868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:04.066839    5419 generic.go:356] "Generic (PLEG): container finished" podID="8adc9451-975f-4d36-986e-97a7f0542554" containerID="9c57725e493ad50062a93c7ff4500a711301a6c621a3f53b944169944dbee274" exitCode=0
Oct 31 01:16:04.067303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:04.066898    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg" event={"ID":"8adc9451-975f-4d36-986e-97a7f0542554","Type":"ContainerDied","Data":"9c57725e493ad50062a93c7ff4500a711301a6c621a3f53b944169944dbee274"}
Oct 31 01:16:04.963856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:04.963818    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"]
Oct 31 01:16:05.209994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.209971    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:05.286912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.286882    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca\") pod \"8adc9451-975f-4d36-986e-97a7f0542554\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") "
Oct 31 01:16:05.287080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.286977    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb\" (UniqueName: \"kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb\") pod \"8adc9451-975f-4d36-986e-97a7f0542554\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") "
Oct 31 01:16:05.287080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.287011    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca\") pod \"8adc9451-975f-4d36-986e-97a7f0542554\" (UID: \"8adc9451-975f-4d36-986e-97a7f0542554\") "
Oct 31 01:16:05.287327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.287303    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca" pod "8adc9451-975f-4d36-986e-97a7f0542554" (UID: "8adc9451-975f-4d36-986e-97a7f0542554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:05.289402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.289370    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb" pod "8adc9451-975f-4d36-986e-97a7f0542554" (UID: "8adc9451-975f-4d36-986e-97a7f0542554"). InnerVolumeSpecName "kube-api-access-b48cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:05.289500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.289400    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca" pod "8adc9451-975f-4d36-986e-97a7f0542554" (UID: "8adc9451-975f-4d36-986e-97a7f0542554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:16:05.388919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.388835    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:05.388919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.388874    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48cb\" (UniqueName: \"kubernetes.io/projected/8adc9451-975f-4d36-986e-97a7f0542554-kube-api-access-b48cb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:05.388919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:05.388891    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8adc9451-975f-4d36-986e-97a7f0542554-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:06.080254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:06.080215    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg" event={"ID":"8adc9451-975f-4d36-986e-97a7f0542554","Type":"ContainerDied","Data":"7dd488cd74acd845ccea581e797fc503aef50a04e17a4f82b62475e1ca1f9f96"}
Oct 31 01:16:06.080254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:06.080239    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"
Oct 31 01:16:06.080524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:06.080280    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dd488cd74acd845ccea581e797fc503aef50a04e17a4f82b62475e1ca1f9f96"
Oct 31 01:16:07.515095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.515054    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv" podUID="45980ec5-7ede-4bfd-b21f-0edbcf91899f" containerName="payload-processing" containerID="cri-o://1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8" gracePeriod=25
Oct 31 01:16:07.749101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.749080    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:16:07.812521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.812436    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm\" (UniqueName: \"kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm\") pod \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") "
Oct 31 01:16:07.812694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.812526    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume\") pod \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\" (UID: \"45980ec5-7ede-4bfd-b21f-0edbcf91899f\") "
Oct 31 01:16:07.812877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.812856    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume" pod "45980ec5-7ede-4bfd-b21f-0edbcf91899f" (UID: "45980ec5-7ede-4bfd-b21f-0edbcf91899f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:07.814710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.814683    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm" pod "45980ec5-7ede-4bfd-b21f-0edbcf91899f" (UID: "45980ec5-7ede-4bfd-b21f-0edbcf91899f"). InnerVolumeSpecName "kube-api-access-shkdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:07.913742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.913700    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shkdm\" (UniqueName: \"kubernetes.io/projected/45980ec5-7ede-4bfd-b21f-0edbcf91899f-kube-api-access-shkdm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:07.913742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:07.913738    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45980ec5-7ede-4bfd-b21f-0edbcf91899f-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:08.092883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.092803    5419 generic.go:356] "Generic (PLEG): container finished" podID="45980ec5-7ede-4bfd-b21f-0edbcf91899f" containerID="1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8" exitCode=0
Oct 31 01:16:08.092883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.092854    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv" event={"ID":"45980ec5-7ede-4bfd-b21f-0edbcf91899f","Type":"ContainerDied","Data":"1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8"}
Oct 31 01:16:08.092883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.092857    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"
Oct 31 01:16:08.093115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.092896    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv" event={"ID":"45980ec5-7ede-4bfd-b21f-0edbcf91899f","Type":"ContainerDied","Data":"fd484fe658a1730d831764e18c9c6db049c3339cd206a01e71d3dbf9ec3ee5ce"}
Oct 31 01:16:08.093115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.092916    5419 scope.go:122] "RemoveContainer" containerID="1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8"
Oct 31 01:16:08.102164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.102129    5419 scope.go:122] "RemoveContainer" containerID="1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8"
Oct 31 01:16:08.102450 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:08.102430    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8\": container with ID starting with 1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8 not found: ID does not exist" containerID="1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8"
Oct 31 01:16:08.102513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.102461    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8"} err="failed to get container status \"1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8\": rpc error: code = NotFound desc = could not find container \"1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8\": container with ID starting with 1034719b5e7ddd43de42e3700c51a605df255d9041697e986d3a8744f874dbb8 not found: ID does not exist"
Oct 31 01:16:08.258244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.258209    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"]
Oct 31 01:16:08.265679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.265652    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7164dd00-6c7b4c868c-6h8bv"]
Oct 31 01:16:08.835786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:08.835755    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45980ec5-7ede-4bfd-b21f-0edbcf91899f" path="/var/lib/kubelet/pods/45980ec5-7ede-4bfd-b21f-0edbcf91899f/volumes"
Oct 31 01:16:09.964236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:09.964192    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9" podUID="f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" containerName="payload-processing" containerID="cri-o://02ec6b66a9874180a3b3a9e186ebf8b26cbe0e16f4a57124e28c0d178fbac068" gracePeriod=25
Oct 31 01:16:10.115648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.115387    5419 generic.go:356] "Generic (PLEG): container finished" podID="f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" containerID="02ec6b66a9874180a3b3a9e186ebf8b26cbe0e16f4a57124e28c0d178fbac068" exitCode=0
Oct 31 01:16:10.115648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.115581    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9" event={"ID":"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46","Type":"ContainerDied","Data":"02ec6b66a9874180a3b3a9e186ebf8b26cbe0e16f4a57124e28c0d178fbac068"}
Oct 31 01:16:10.229257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.229233    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:16:10.239207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.239188    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume\") pod \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") "
Oct 31 01:16:10.239313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.239247    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm\" (UniqueName: \"kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm\") pod \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\" (UID: \"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46\") "
Oct 31 01:16:10.239562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.239528    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume" pod "f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" (UID: "f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:10.241347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.241325    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm" pod "f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" (UID: "f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46"). InnerVolumeSpecName "kube-api-access-zc5zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:10.355162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.354498    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:16:10.357406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.357081    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:10.357406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:10.357115    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc5zm\" (UniqueName: \"kubernetes.io/projected/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46-kube-api-access-zc5zm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:11.122045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:11.121965    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"
Oct 31 01:16:11.122045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:11.121996    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9" event={"ID":"f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46","Type":"ContainerDied","Data":"bd8a0e48ecc23ca87feedc1385e2d47651073fad981f192a1bdecdff25862d2a"}
Oct 31 01:16:11.122584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:11.122052    5419 scope.go:122] "RemoveContainer" containerID="02ec6b66a9874180a3b3a9e186ebf8b26cbe0e16f4a57124e28c0d178fbac068"
Oct 31 01:16:11.284587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:11.284553    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"]
Oct 31 01:16:11.333718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:11.333685    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b9656f65-79bd98bcf7-fwcj9"]
Oct 31 01:16:12.835600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:12.835568    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" path="/var/lib/kubelet/pods/f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46/volumes"
Oct 31 01:16:13.620239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:13.620203    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"]
Oct 31 01:16:13.668779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:13.668748    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b9656f65-fth8z"]
Oct 31 01:16:14.581572 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:14.581534    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:16:14.774914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:14.774875    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"]
Oct 31 01:16:14.836727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:14.836657    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0157f954-14b1-45f5-8042-09e7723d3c86" path="/var/lib/kubelet/pods/0157f954-14b1-45f5-8042-09e7723d3c86/volumes"
Oct 31 01:16:14.887198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:14.887163    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"]
Oct 31 01:16:15.044703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.044672    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"]
Oct 31 01:16:15.045071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045059    5419 cpu_manager.go:414] "containerMap: removing container" podUID="45980ec5-7ede-4bfd-b21f-0edbcf91899f" containerName="payload-processing"
Oct 31 01:16:15.045127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045073    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45980ec5-7ede-4bfd-b21f-0edbcf91899f" containerName="payload-processing"
Oct 31 01:16:15.045127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045091    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8adc9451-975f-4d36-986e-97a7f0542554" containerName="revoke-keys"
Oct 31 01:16:15.045127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045096    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8adc9451-975f-4d36-986e-97a7f0542554" containerName="revoke-keys"
Oct 31 01:16:15.045127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045119    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" containerName="payload-processing"
Oct 31 01:16:15.045127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045124    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" containerName="payload-processing"
Oct 31 01:16:15.045313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045209    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f219f56b-b1f7-4fa1-bf9c-cb95c2d5cb46" containerName="payload-processing"
Oct 31 01:16:15.045313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045221    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="45980ec5-7ede-4bfd-b21f-0edbcf91899f" containerName="payload-processing"
Oct 31 01:16:15.045313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.045233    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8adc9451-975f-4d36-986e-97a7f0542554" containerName="revoke-keys"
Oct 31 01:16:15.051456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.051424    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.098831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.098743    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.098831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.098804    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.098900    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.098933    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfnh\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.098965    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.099095    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.099170    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.099277    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.099351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.099337    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.103596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.103571    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-baacee-openshift-default-dockercfg-pr4tt\""
Oct 31 01:16:15.110320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.110292    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"]
Oct 31 01:16:15.202983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.202947    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.202983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.202995    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203021    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfnh\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203060    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203103    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203132    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203246    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203304    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.203375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.203350    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.205392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.205355    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.205680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.205655    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.214886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.206413    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.214886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.210714    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.214886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.212347    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.214886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.213740    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.239015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.238980    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.269493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.269465    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.269630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.269533    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfnh\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.365565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.365465    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:15.865081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.865049    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"]
Oct 31 01:16:15.868213 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:15.868184    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3bb1ca_6e4a_465f_bb37_18141137d234.slice/crio-d119d0f1b38f5cd1a9f171eb88982b1fb5cbc78fed14e8bc6f12eab502db8026 WatchSource:0}: Error finding container d119d0f1b38f5cd1a9f171eb88982b1fb5cbc78fed14e8bc6f12eab502db8026: Status 404 returned error can't find the container with id d119d0f1b38f5cd1a9f171eb88982b1fb5cbc78fed14e8bc6f12eab502db8026
Oct 31 01:16:15.870755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.870725    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:15.870827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.870791    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:15.870827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:15.870817    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:16.150043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:16.149958    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" event={"ID":"cb3bb1ca-6e4a-465f-bb37-18141137d234","Type":"ContainerStarted","Data":"c2a9da38006f89c7e4d3e7bc8f0803ab6ab38dae1d395d3776d5f8255ee3a02f"}
Oct 31 01:16:16.150043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:16.150011    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" event={"ID":"cb3bb1ca-6e4a-465f-bb37-18141137d234","Type":"ContainerStarted","Data":"d119d0f1b38f5cd1a9f171eb88982b1fb5cbc78fed14e8bc6f12eab502db8026"}
Oct 31 01:16:16.268126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:16.268067    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podStartSLOduration=1.2680493099999999 podStartE2EDuration="1.26804931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:16.267490847 +0000 UTC m=+697.568447426" watchObservedRunningTime="2026-10-31 01:16:16.26804931 +0000 UTC m=+697.569005888"
Oct 31 01:16:16.366039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:16.366004    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:16.371440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:16.371412    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:17.155091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:17.155065    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:17.155998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:17.155979    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:19.245007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:19.244963    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:16:19.306091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:19.306054    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"]
Oct 31 01:16:19.308497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:19.308473    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f94871a1-79xh8"]
Oct 31 01:16:19.776059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:19.775999    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf" podUID="1215c783-ea3f-48af-b15d-5494467badca" containerName="payload-processing" containerID="cri-o://5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f" gracePeriod=25
Oct 31 01:16:19.888434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:19.888361    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj" podUID="afaf2744-8709-4540-962b-a5de808bc38a" containerName="payload-pre-processing" containerID="cri-o://6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9" gracePeriod=25
Oct 31 01:16:20.036538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.036482    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:16:20.151064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.151040    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:16:20.171912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.171885    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj\" (UniqueName: \"kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj\") pod \"1215c783-ea3f-48af-b15d-5494467badca\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") "
Oct 31 01:16:20.172094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.171964    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume\") pod \"1215c783-ea3f-48af-b15d-5494467badca\" (UID: \"1215c783-ea3f-48af-b15d-5494467badca\") "
Oct 31 01:16:20.172383 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.172358    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume" pod "1215c783-ea3f-48af-b15d-5494467badca" (UID: "1215c783-ea3f-48af-b15d-5494467badca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:20.174298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.174277    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj" pod "1215c783-ea3f-48af-b15d-5494467badca" (UID: "1215c783-ea3f-48af-b15d-5494467badca"). InnerVolumeSpecName "kube-api-access-hpdmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:20.177594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.177570    5419 generic.go:356] "Generic (PLEG): container finished" podID="1215c783-ea3f-48af-b15d-5494467badca" containerID="5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f" exitCode=0
Oct 31 01:16:20.177695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.177616    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"
Oct 31 01:16:20.177695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.177652    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf" event={"ID":"1215c783-ea3f-48af-b15d-5494467badca","Type":"ContainerDied","Data":"5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f"}
Oct 31 01:16:20.177792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.177709    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf" event={"ID":"1215c783-ea3f-48af-b15d-5494467badca","Type":"ContainerDied","Data":"d3712fa6aaea961e88f4f237836bdb822035b2ad71cec035cb0daf38cf55c830"}
Oct 31 01:16:20.177792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.177731    5419 scope.go:122] "RemoveContainer" containerID="5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f"
Oct 31 01:16:20.182393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.182369    5419 generic.go:356] "Generic (PLEG): container finished" podID="afaf2744-8709-4540-962b-a5de808bc38a" containerID="6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9" exitCode=0
Oct 31 01:16:20.182519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.182432    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"
Oct 31 01:16:20.182519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.182443    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj" event={"ID":"afaf2744-8709-4540-962b-a5de808bc38a","Type":"ContainerDied","Data":"6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9"}
Oct 31 01:16:20.182519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.182492    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj" event={"ID":"afaf2744-8709-4540-962b-a5de808bc38a","Type":"ContainerDied","Data":"7ce6fe8f4a00c286a5444ade84276e06e8dd6486221e921449c32322dcc1da57"}
Oct 31 01:16:20.192308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.192282    5419 scope.go:122] "RemoveContainer" containerID="5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f"
Oct 31 01:16:20.192633 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:20.192606    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f\": container with ID starting with 5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f not found: ID does not exist" containerID="5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f"
Oct 31 01:16:20.192738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.192640    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f"} err="failed to get container status \"5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f\": rpc error: code = NotFound desc = could not find container \"5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f\": container with ID starting with 5a31f4640606703064d9cb46fca8fafc3d515945df49ab98959fd6a33022f86f not found: ID does not exist"
Oct 31 01:16:20.192738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.192658    5419 scope.go:122] "RemoveContainer" containerID="6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9"
Oct 31 01:16:20.201203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.201185    5419 scope.go:122] "RemoveContainer" containerID="6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9"
Oct 31 01:16:20.201502 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:20.201480    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9\": container with ID starting with 6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9 not found: ID does not exist" containerID="6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9"
Oct 31 01:16:20.201580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.201515    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9"} err="failed to get container status \"6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9\": rpc error: code = NotFound desc = could not find container \"6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9\": container with ID starting with 6109e1f551a3642fd81361b117c4a2ca0bf0e3c30d8444a5f70586e357d60ae9 not found: ID does not exist"
Oct 31 01:16:20.273482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.273445    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume\") pod \"afaf2744-8709-4540-962b-a5de808bc38a\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") "
Oct 31 01:16:20.273920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.273509    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb\" (UniqueName: \"kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb\") pod \"afaf2744-8709-4540-962b-a5de808bc38a\" (UID: \"afaf2744-8709-4540-962b-a5de808bc38a\") "
Oct 31 01:16:20.273920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.273897    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpdmj\" (UniqueName: \"kubernetes.io/projected/1215c783-ea3f-48af-b15d-5494467badca-kube-api-access-hpdmj\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:20.274061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.273928    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1215c783-ea3f-48af-b15d-5494467badca-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:20.274061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.273893    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume" pod "afaf2744-8709-4540-962b-a5de808bc38a" (UID: "afaf2744-8709-4540-962b-a5de808bc38a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:20.275872 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.275850    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb" pod "afaf2744-8709-4540-962b-a5de808bc38a" (UID: "afaf2744-8709-4540-962b-a5de808bc38a"). InnerVolumeSpecName "kube-api-access-sdqgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:20.374950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.374861    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afaf2744-8709-4540-962b-a5de808bc38a-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:20.374950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.374896    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdqgb\" (UniqueName: \"kubernetes.io/projected/afaf2744-8709-4540-962b-a5de808bc38a-kube-api-access-sdqgb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:20.428028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.427999    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"]
Oct 31 01:16:20.431576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.431552    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-870eefeb-6f5465c587-ppmxf"]
Oct 31 01:16:20.602725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.602695    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"]
Oct 31 01:16:20.606432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.606408    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-870eefeb-565c4c4655-tx5wj"]
Oct 31 01:16:20.840704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.840671    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1215c783-ea3f-48af-b15d-5494467badca" path="/var/lib/kubelet/pods/1215c783-ea3f-48af-b15d-5494467badca/volumes"
Oct 31 01:16:20.841269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.841252    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322704de-c256-47ff-b401-6e193777abac" path="/var/lib/kubelet/pods/322704de-c256-47ff-b401-6e193777abac/volumes"
Oct 31 01:16:20.841741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.841724    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afaf2744-8709-4540-962b-a5de808bc38a" path="/var/lib/kubelet/pods/afaf2744-8709-4540-962b-a5de808bc38a/volumes"
Oct 31 01:16:20.842081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.842065    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"]
Oct 31 01:16:20.842363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:20.842338    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy" containerID="cri-o://cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3" gracePeriod=30
Oct 31 01:16:23.513576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.513543    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"]
Oct 31 01:16:23.514094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.514063    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1215c783-ea3f-48af-b15d-5494467badca" containerName="payload-processing"
Oct 31 01:16:23.514094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.514082    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1215c783-ea3f-48af-b15d-5494467badca" containerName="payload-processing"
Oct 31 01:16:23.514269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.514110    5419 cpu_manager.go:414] "containerMap: removing container" podUID="afaf2744-8709-4540-962b-a5de808bc38a" containerName="payload-pre-processing"
Oct 31 01:16:23.514269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.514118    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afaf2744-8709-4540-962b-a5de808bc38a" containerName="payload-pre-processing"
Oct 31 01:16:23.514269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.514249    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="afaf2744-8709-4540-962b-a5de808bc38a" containerName="payload-pre-processing"
Oct 31 01:16:23.514269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.514270    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1215c783-ea3f-48af-b15d-5494467badca" containerName="payload-processing"
Oct 31 01:16:23.519079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.519047    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.606340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.606310    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.606530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.606507    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxcgq\" (UniqueName: \"kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq\") pod \"payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.622123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.622087    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-baacee-dockercfg-f62tl\""
Oct 31 01:16:23.622299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.622187    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-baacee\""
Oct 31 01:16:23.623769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.623750    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"]
Oct 31 01:16:23.708022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.707986    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxcgq\" (UniqueName: \"kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq\") pod \"payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.708243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.708055    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.708683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.708663    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.765350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.765276    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxcgq\" (UniqueName: \"kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq\") pod \"payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:23.834248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:23.832682    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:24.213748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:24.213672    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"]
Oct 31 01:16:24.216404 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:24.216377    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ade45e_da1d_4a36_ba56_228c267a8fa4.slice/crio-84783eb91c51b82c9d156ddfe93660509a8d34337e86cedfd1798f20e39da0fa WatchSource:0}: Error finding container 84783eb91c51b82c9d156ddfe93660509a8d34337e86cedfd1798f20e39da0fa: Status 404 returned error can't find the container with id 84783eb91c51b82c9d156ddfe93660509a8d34337e86cedfd1798f20e39da0fa
Oct 31 01:16:25.210460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:25.210427    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" event={"ID":"e7ade45e-da1d-4a36-ba56-228c267a8fa4","Type":"ContainerStarted","Data":"7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362"}
Oct 31 01:16:25.210906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:25.210477    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" event={"ID":"e7ade45e-da1d-4a36-ba56-228c267a8fa4","Type":"ContainerStarted","Data":"84783eb91c51b82c9d156ddfe93660509a8d34337e86cedfd1798f20e39da0fa"}
Oct 31 01:16:25.210906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:25.210550    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:25.326053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:25.326000    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" podStartSLOduration=2.325987734 podStartE2EDuration="2.325987734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:25.324758058 +0000 UTC m=+706.625714632" watchObservedRunningTime="2026-10-31 01:16:25.325987734 +0000 UTC m=+706.626944311"
Oct 31 01:16:25.782068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:25.782028    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"]
Oct 31 01:16:25.784246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:25.784220    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7164dd00-79k2g"]
Oct 31 01:16:26.087317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.087292    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:16:26.218598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.218560    5419 generic.go:356] "Generic (PLEG): container finished" podID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerID="cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3" exitCode=0
Oct 31 01:16:26.219058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.218629    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"
Oct 31 01:16:26.219058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.218641    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" event={"ID":"00630b5c-589c-4b00-b8c2-e4055c355c9f","Type":"ContainerDied","Data":"cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3"}
Oct 31 01:16:26.219058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.218704    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz" event={"ID":"00630b5c-589c-4b00-b8c2-e4055c355c9f","Type":"ContainerDied","Data":"321dc656c3a899ec639e9111b1a4eb3348853515b122337e1f29c9d6ee9f481a"}
Oct 31 01:16:26.219058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.218737    5419 scope.go:122] "RemoveContainer" containerID="cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3"
Oct 31 01:16:26.228023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.228001    5419 scope.go:122] "RemoveContainer" containerID="d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"
Oct 31 01:16:26.235415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235388    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235424    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235469    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235500    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235548    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235609    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235642    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235687    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235740    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert\") pod \"00630b5c-589c-4b00-b8c2-e4055c355c9f\" (UID: \"00630b5c-589c-4b00-b8c2-e4055c355c9f\") "
Oct 31 01:16:26.235849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.235837    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:26.236313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.236174    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.236313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.236239    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:26.236433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.236334    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:26.236433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.236406    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:26.236614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.236584    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:26.237514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.237493    5419 scope.go:122] "RemoveContainer" containerID="cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3"
Oct 31 01:16:26.237986 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:26.237816    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3\": container with ID starting with cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3 not found: ID does not exist" containerID="cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3"
Oct 31 01:16:26.237986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.237858    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3"} err="failed to get container status \"cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3\": rpc error: code = NotFound desc = could not find container \"cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3\": container with ID starting with cff9415f8a676c52d8d463d303ec5a20ae36cae8fdace06cea226e4c2a3e8dd3 not found: ID does not exist"
Oct 31 01:16:26.237986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.237885    5419 scope.go:122] "RemoveContainer" containerID="d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"
Oct 31 01:16:26.238279 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:26.238238    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826\": container with ID starting with d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826 not found: ID does not exist" containerID="d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"
Oct 31 01:16:26.238279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.238267    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826"} err="failed to get container status \"d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826\": rpc error: code = NotFound desc = could not find container \"d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826\": container with ID starting with d4a73621501279cc59753d869517f53bfeef9fccb7bcdc26ce6917aaa96bd826 not found: ID does not exist"
Oct 31 01:16:26.238519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.238497    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:26.238611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.238587    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:16:26.238858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.238832    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:26.238858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.238846    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7" pod "00630b5c-589c-4b00-b8c2-e4055c355c9f" (UID: "00630b5c-589c-4b00-b8c2-e4055c355c9f"). InnerVolumeSpecName "kube-api-access-c5kl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:26.336850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336785    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.336850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336810    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.336850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336826    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00630b5c-589c-4b00-b8c2-e4055c355c9f-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.336850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336836    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.336850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336844    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.336850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336853    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5kl7\" (UniqueName: \"kubernetes.io/projected/00630b5c-589c-4b00-b8c2-e4055c355c9f-kube-api-access-c5kl7\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.337146 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336861    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00630b5c-589c-4b00-b8c2-e4055c355c9f-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.337146 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.336869    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00630b5c-589c-4b00-b8c2-e4055c355c9f-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:26.639818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.639728    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"]
Oct 31 01:16:26.644695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.644668    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f94871a1-openshift-default-785fbb566c-g9qtz"]
Oct 31 01:16:26.841214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.837971    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" path="/var/lib/kubelet/pods/00630b5c-589c-4b00-b8c2-e4055c355c9f/volumes"
Oct 31 01:16:26.841214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:26.838628    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d873d741-d410-4f63-8c5c-c7b62d8d3b4e" path="/var/lib/kubelet/pods/d873d741-d410-4f63-8c5c-c7b62d8d3b4e/volumes"
Oct 31 01:16:27.073638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:27.073605    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"]
Oct 31 01:16:27.077031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:27.077008    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-870eefeb-n77sg"]
Oct 31 01:16:27.116277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:27.116247    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"]
Oct 31 01:16:27.116478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:27.116452    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" containerID="cri-o://44780bafa93f934b0e3e51356af89d7580dae7357bfe8d5b32b17dc81ebc22c4" gracePeriod=30
Oct 31 01:16:28.659207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:28.659168    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:16:28.835525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:28.835497    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8adc9451-975f-4d36-986e-97a7f0542554" path="/var/lib/kubelet/pods/8adc9451-975f-4d36-986e-97a7f0542554/volumes"
Oct 31 01:16:29.300523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:29.300490    5419 patch_prober.go:29] interesting pod/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:16:29.300700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:29.300547    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:16:32.252401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.252351    5419 generic.go:356] "Generic (PLEG): container finished" podID="26586095-56c1-4668-a769-0b96f2aecf4f" containerID="44780bafa93f934b0e3e51356af89d7580dae7357bfe8d5b32b17dc81ebc22c4" exitCode=0
Oct 31 01:16:32.252908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.252415    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" event={"ID":"26586095-56c1-4668-a769-0b96f2aecf4f","Type":"ContainerDied","Data":"44780bafa93f934b0e3e51356af89d7580dae7357bfe8d5b32b17dc81ebc22c4"}
Oct 31 01:16:32.252908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.252514    5419 scope.go:122] "RemoveContainer" containerID="ae3a066f7246a45efc970e4a7104cb054627800a214933a43ad4b69f187aa2d0"
Oct 31 01:16:32.420974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.420642    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:16:32.507280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507244    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507294    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507318    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507353    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507384    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507438    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507459    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507527    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507555    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data\") pod \"26586095-56c1-4668-a769-0b96f2aecf4f\" (UID: \"26586095-56c1-4668-a769-0b96f2aecf4f\") "
Oct 31 01:16:32.507770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507625    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:32.507770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507692    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:32.507984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507855    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.507984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507869    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.507984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507962    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:32.507984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.507972    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:32.508298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.508268    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:32.509829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.509799    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:16:32.509945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.509868    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:32.510276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.510258    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:32.510335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.510312    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9" pod "26586095-56c1-4668-a769-0b96f2aecf4f" (UID: "26586095-56c1-4668-a769-0b96f2aecf4f"). InnerVolumeSpecName "kube-api-access-nvbx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:32.608769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608733    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.608769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608766    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26586095-56c1-4668-a769-0b96f2aecf4f-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.608769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608776    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbx9\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-kube-api-access-nvbx9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.608994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608786    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26586095-56c1-4668-a769-0b96f2aecf4f-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.608994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608795    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.608994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608804    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26586095-56c1-4668-a769-0b96f2aecf4f-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:32.608994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:32.608811    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26586095-56c1-4668-a769-0b96f2aecf4f-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:33.259410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:33.259381    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"
Oct 31 01:16:33.259410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:33.259387    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w" event={"ID":"26586095-56c1-4668-a769-0b96f2aecf4f","Type":"ContainerDied","Data":"a8b070d82df52a74ae4dfa66d9a593854bcff4d23534df7b5da48e223feba2bc"}
Oct 31 01:16:33.259921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:33.259458    5419 scope.go:122] "RemoveContainer" containerID="44780bafa93f934b0e3e51356af89d7580dae7357bfe8d5b32b17dc81ebc22c4"
Oct 31 01:16:33.471246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:33.471216    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"]
Oct 31 01:16:33.520816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:33.520740    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-870eefeb-openshift-default-6b9747f565-75x6w"]
Oct 31 01:16:34.836809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:34.836774    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" path="/var/lib/kubelet/pods/26586095-56c1-4668-a769-0b96f2aecf4f/volumes"
Oct 31 01:16:36.221976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:36.221950    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 01:16:36.401165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:36.390601    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"]
Oct 31 01:16:36.401510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:36.401464    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy" containerID="cri-o://54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f" gracePeriod=30
Oct 31 01:16:37.703479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:37.703398    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:16:39.041263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.041230    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:16:39.041634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.041276    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:16:39.395575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.395538    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"]
Oct 31 01:16:39.396379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.396356    5419 cpu_manager.go:414] "containerMap: removing container" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy"
Oct 31 01:16:39.396502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.396489    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy"
Oct 31 01:16:39.396593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.396583    5419 cpu_manager.go:414] "containerMap: removing container" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy"
Oct 31 01:16:39.396666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.396656    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy"
Oct 31 01:16:39.396852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.396841    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy"
Oct 31 01:16:39.396952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.396942    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy"
Oct 31 01:16:39.397039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.397030    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy"
Oct 31 01:16:39.407197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.407170    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.450663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.450634    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"]
Oct 31 01:16:39.503525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.503490    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"openshift-service-ca.crt\""
Oct 31 01:16:39.503730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.503664    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"simulator-w3-a8e157-kserve-self-signed-certs\""
Oct 31 01:16:39.508999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.508965    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"default-dockercfg-rlpbk\""
Oct 31 01:16:39.509120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.509028    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a8e157\"/\"kube-root-ca.crt\""
Oct 31 01:16:39.572283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.572249    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5sn\" (UniqueName: \"kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.572283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.572284    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.572510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.572356    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.572510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.572375    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.572585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.572516    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.572585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.572579    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.673638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673555    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.673638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673595    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.673638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673621    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.673918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673664    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.673918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673733    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5sn\" (UniqueName: \"kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.673918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673764    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.674102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.673997    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.674102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.674051    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.674256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.674132    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.699168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.695409    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.699168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.696785    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:39.731379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:39.731342    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5sn\" (UniqueName: \"kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn\") pod \"simulator-w3-a8e157-kserve-5c554d85ff-8v2gm\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") " pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:40.022219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.021990    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:40.421670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.420792    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"]
Oct 31 01:16:40.424416 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:40.424377    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43590537_8e3d_459c_a696_0222664cf92d.slice/crio-1374dd9a31016fbc277a6c49ecadc0c0390f298defa50305fdfd981876c35a75 WatchSource:0}: Error finding container 1374dd9a31016fbc277a6c49ecadc0c0390f298defa50305fdfd981876c35a75: Status 404 returned error can't find the container with id 1374dd9a31016fbc277a6c49ecadc0c0390f298defa50305fdfd981876c35a75
Oct 31 01:16:40.535624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.535584    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"]
Oct 31 01:16:40.535994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.535979    5419 cpu_manager.go:414] "containerMap: removing container" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy"
Oct 31 01:16:40.535994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.535994    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00630b5c-589c-4b00-b8c2-e4055c355c9f" containerName="istio-proxy"
Oct 31 01:16:40.536125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.536012    5419 cpu_manager.go:414] "containerMap: removing container" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy"
Oct 31 01:16:40.536125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.536017    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy"
Oct 31 01:16:40.536125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.536086    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="26586095-56c1-4668-a769-0b96f2aecf4f" containerName="istio-proxy"
Oct 31 01:16:40.539094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.539074    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.598240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.598200    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-00e840\""
Oct 31 01:16:40.611742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.608794    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-00e840-dockercfg-mjhxp\""
Oct 31 01:16:40.611742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.609975    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"]
Oct 31 01:16:40.655654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.655622    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"]
Oct 31 01:16:40.659219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.659197    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.708544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.708457    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.708544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.708523    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpv2\" (UniqueName: \"kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2\") pod \"payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.717846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.717816    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"]
Oct 31 01:16:40.809306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.809264    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.809485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.809326    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.809485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.809429    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpv2\" (UniqueName: \"kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2\") pod \"payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.809485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.809468    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjz9\" (UniqueName: \"kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9\") pod \"payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.810361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.810338    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.865935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.865893    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpv2\" (UniqueName: \"kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2\") pod \"payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:40.910800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.910762    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.913197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.910859    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjz9\" (UniqueName: \"kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9\") pod \"payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.913197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.911550    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.967997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.967925    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjz9\" (UniqueName: \"kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9\") pod \"payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:40.971782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:40.971747    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:41.152920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.152840    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:41.330847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.330810    5419 generic.go:356] "Generic (PLEG): container finished" podID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerID="8b8c5a2a157745558a1b41f3cab5f9c4a1033262e5655c84f418743c7643110e" exitCode=0
Oct 31 01:16:41.331021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.330882    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" event={"ID":"ad5493b4-7690-4d3a-8b69-4f8105e82abc","Type":"ContainerDied","Data":"8b8c5a2a157745558a1b41f3cab5f9c4a1033262e5655c84f418743c7643110e"}
Oct 31 01:16:41.331329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.331314    5419 scope.go:122] "RemoveContainer" containerID="8b8c5a2a157745558a1b41f3cab5f9c4a1033262e5655c84f418743c7643110e"
Oct 31 01:16:41.332625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.332601    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:41.332754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.332655    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" event={"ID":"43590537-8e3d-459c-a696-0222664cf92d","Type":"ContainerStarted","Data":"b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63"}
Oct 31 01:16:41.332754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.332681    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" event={"ID":"43590537-8e3d-459c-a696-0222664cf92d","Type":"ContainerStarted","Data":"1374dd9a31016fbc277a6c49ecadc0c0390f298defa50305fdfd981876c35a75"}
Oct 31 01:16:41.332754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.332689    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:41.332754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.332737    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:41.385446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.374738    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"]
Oct 31 01:16:41.552775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.550815    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"]
Oct 31 01:16:41.554014 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:41.553986    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c9a7d8_a8d6_42d9_abbe_96d39977d4e3.slice/crio-22f224491139bbe568afc08c29c13dbedcd0bf28e9ea0c8801448fa435f0cb91 WatchSource:0}: Error finding container 22f224491139bbe568afc08c29c13dbedcd0bf28e9ea0c8801448fa435f0cb91: Status 404 returned error can't find the container with id 22f224491139bbe568afc08c29c13dbedcd0bf28e9ea0c8801448fa435f0cb91
Oct 31 01:16:41.588161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.586253    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podStartSLOduration=2.586226931 podStartE2EDuration="2.586226931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:41.581311848 +0000 UTC m=+722.882268426" watchObservedRunningTime="2026-10-31 01:16:41.586226931 +0000 UTC m=+722.887183535"
Oct 31 01:16:41.749237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.749212    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924608    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924695    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924733    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924788    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924823    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924861    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924898    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.924946    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.925170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.925018    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token\") pod \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\" (UID: \"63ca9a39-e693-418a-b9ed-5f64ac4475b4\") "
Oct 31 01:16:41.927528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.926579    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:41.927687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.927590    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:41.928897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.928740    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:41.928897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.928855    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:41.929107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.928948    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:41.931336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.931313    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:16:41.931647 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.931620    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "kube-api-access-h76rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:41.932916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.932894    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:41.933407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:41.933388    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy" pod "63ca9a39-e693-418a-b9ed-5f64ac4475b4" (UID: "63ca9a39-e693-418a-b9ed-5f64ac4475b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:42.029222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.029017    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.029981    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h76rj\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-kube-api-access-h76rj\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030004    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030013    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030023    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030033    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030042    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030051    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030059    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ca9a39-e693-418a-b9ed-5f64ac4475b4-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.030169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.030066    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ca9a39-e693-418a-b9ed-5f64ac4475b4-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:16:42.338847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.338754    5419 generic.go:356] "Generic (PLEG): container finished" podID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerID="54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f" exitCode=0
Oct 31 01:16:42.338847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.338804    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"
Oct 31 01:16:42.338847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.338832    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" event={"ID":"63ca9a39-e693-418a-b9ed-5f64ac4475b4","Type":"ContainerDied","Data":"54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f"}
Oct 31 01:16:42.339098 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.338887    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5" event={"ID":"63ca9a39-e693-418a-b9ed-5f64ac4475b4","Type":"ContainerDied","Data":"c907f0e22d9d4e4c6a55f473dd26b8eda23cd84daeb61bc0cd31a0ba6347123d"}
Oct 31 01:16:42.339098 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.338914    5419 scope.go:122] "RemoveContainer" containerID="54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f"
Oct 31 01:16:42.341249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.341226    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2" event={"ID":"fd6684c2-8011-4447-855c-7671fbc0a984","Type":"ContainerStarted","Data":"b18aa6cdb4bdb0c502ac90a8e63d00266b95274023a456659ed0bdf4b919216a"}
Oct 31 01:16:42.341331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.341276    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2" event={"ID":"fd6684c2-8011-4447-855c-7671fbc0a984","Type":"ContainerStarted","Data":"c6a472b13be5f534ccbc288aa50b2f9464d4c203650059069f9413e431a9ccaa"}
Oct 31 01:16:42.341403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.341390    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:42.343521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.343503    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" event={"ID":"ad5493b4-7690-4d3a-8b69-4f8105e82abc","Type":"ContainerStarted","Data":"3ad73901e95a45ec3ddd583bf3b2be84a86023d4c91a590e718a5dbef00e0937"}
Oct 31 01:16:42.346183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.345700    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" event={"ID":"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3","Type":"ContainerStarted","Data":"622437b3af648ee8321100ade2e04c9d45064f385454ccf9b31299160a777096"}
Oct 31 01:16:42.346183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.345740    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" event={"ID":"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3","Type":"ContainerStarted","Data":"22f224491139bbe568afc08c29c13dbedcd0bf28e9ea0c8801448fa435f0cb91"}
Oct 31 01:16:42.346183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.345764    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:42.349949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.349933    5419 scope.go:122] "RemoveContainer" containerID="9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"
Oct 31 01:16:42.357741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.357726    5419 scope.go:122] "RemoveContainer" containerID="54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f"
Oct 31 01:16:42.358008 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:42.357989    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f\": container with ID starting with 54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f not found: ID does not exist" containerID="54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f"
Oct 31 01:16:42.358078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.358019    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f"} err="failed to get container status \"54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f\": rpc error: code = NotFound desc = could not find container \"54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f\": container with ID starting with 54341ad9287e7d2085ce078164a697944b9550f7d4ea8034850293e9171c1e2f not found: ID does not exist"
Oct 31 01:16:42.358078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.358041    5419 scope.go:122] "RemoveContainer" containerID="9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"
Oct 31 01:16:42.358304 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:42.358285    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466\": container with ID starting with 9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466 not found: ID does not exist" containerID="9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"
Oct 31 01:16:42.358348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.358311    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466"} err="failed to get container status \"9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466\": rpc error: code = NotFound desc = could not find container \"9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466\": container with ID starting with 9842e7b5e4ab4511d7fc334dca8bdb4e909f51152f087b2a5041c5388be6f466 not found: ID does not exist"
Oct 31 01:16:42.402161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.401492    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:16:42.405165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.403602    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:16:42.405165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.403677    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 01:16:42.629460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.625738    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" podStartSLOduration=2.625717935 podStartE2EDuration="2.625717935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:42.512540304 +0000 UTC m=+723.813496881" watchObservedRunningTime="2026-10-31 01:16:42.625717935 +0000 UTC m=+723.926674517"
Oct 31 01:16:42.630222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.629601    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"]
Oct 31 01:16:42.630222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.629770    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7164dd00-openshift-default-75654487cc-bd9c5"]
Oct 31 01:16:42.757885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.757833    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2" podStartSLOduration=2.7578179350000003 podStartE2EDuration="2.757817935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:42.756368637 +0000 UTC m=+724.057325225" watchObservedRunningTime="2026-10-31 01:16:42.757817935 +0000 UTC m=+724.058774491"
Oct 31 01:16:42.842208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:42.842007    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" path="/var/lib/kubelet/pods/63ca9a39-e693-418a-b9ed-5f64ac4475b4/volumes"
Oct 31 01:16:43.402163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:43.401569    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:16:43.402163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:43.401634    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 01:16:44.401588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:44.401545    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:16:44.402119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:44.401614    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 01:16:44.612195 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:16:44.612159    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:16:45.287165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.282929    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"]
Oct 31 01:16:45.287165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.283515    5419 cpu_manager.go:414] "containerMap: removing container" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy"
Oct 31 01:16:45.287165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.283533    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy"
Oct 31 01:16:45.287165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.283690    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy"
Oct 31 01:16:45.305199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.305163    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.350490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.350436    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"]
Oct 31 01:16:45.365768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.365730    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.366074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.366049    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv\") pod \"payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.407448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.407420    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8f6b5b-dockercfg-9pn6m\""
Oct 31 01:16:45.408435 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.407928    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8f6b5b\""
Oct 31 01:16:45.408742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.408715    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:16:45.408831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.408780    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 01:16:45.418316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.418288    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"]
Oct 31 01:16:45.418733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.418717    5419 cpu_manager.go:414] "containerMap: removing container" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy"
Oct 31 01:16:45.418733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.418732    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy"
Oct 31 01:16:45.418869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.418812    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ca9a39-e693-418a-b9ed-5f64ac4475b4" containerName="istio-proxy"
Oct 31 01:16:45.421962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.421940    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.466995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.466965    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdcb\" (UniqueName: \"kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb\") pod \"payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.467167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.467087    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.467167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.467133    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv\") pod \"payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.467291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.467174    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.467871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.467848    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.499847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.499738    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"]
Oct 31 01:16:45.525546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.525505    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv\") pod \"payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.567845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.567765    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.567845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.567810    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdcb\" (UniqueName: \"kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb\") pod \"payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.568517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.568491    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.620171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.620029    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:45.626250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.626217    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdcb\" (UniqueName: \"kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb\") pod \"payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:45.742705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:45.740330    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:46.031333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.031246    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"]
Oct 31 01:16:46.035391 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:46.035357    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055500fd_ba69_46f9_9824_2ebf5ee9f330.slice/crio-6e040e571c34561988720f325236a07d6fa81d1e042f41ae734bef797851f3a3 WatchSource:0}: Error finding container 6e040e571c34561988720f325236a07d6fa81d1e042f41ae734bef797851f3a3: Status 404 returned error can't find the container with id 6e040e571c34561988720f325236a07d6fa81d1e042f41ae734bef797851f3a3
Oct 31 01:16:46.301868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.301842    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"]
Oct 31 01:16:46.304005 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:46.303971    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e8a15c_fa4c_43ef_a202_3591dd118181.slice/crio-b9942fbf8a51153e01c34c2585a5b6ac45b458c4d5be384a4938f49deeee5e00 WatchSource:0}: Error finding container b9942fbf8a51153e01c34c2585a5b6ac45b458c4d5be384a4938f49deeee5e00: Status 404 returned error can't find the container with id b9942fbf8a51153e01c34c2585a5b6ac45b458c4d5be384a4938f49deeee5e00
Oct 31 01:16:46.401688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.401508    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:16:46.401688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.401563    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 01:16:46.403330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.403297    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" event={"ID":"19e8a15c-fa4c-43ef-a202-3591dd118181","Type":"ContainerStarted","Data":"10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e"}
Oct 31 01:16:46.403466 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.403362    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" event={"ID":"19e8a15c-fa4c-43ef-a202-3591dd118181","Type":"ContainerStarted","Data":"b9942fbf8a51153e01c34c2585a5b6ac45b458c4d5be384a4938f49deeee5e00"}
Oct 31 01:16:46.403531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.403504    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:46.405056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.405031    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" event={"ID":"055500fd-ba69-46f9-9824-2ebf5ee9f330","Type":"ContainerStarted","Data":"26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15"}
Oct 31 01:16:46.405196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.405073    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" event={"ID":"055500fd-ba69-46f9-9824-2ebf5ee9f330","Type":"ContainerStarted","Data":"6e040e571c34561988720f325236a07d6fa81d1e042f41ae734bef797851f3a3"}
Oct 31 01:16:46.405196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.405117    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:46.519915 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.519713    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" podStartSLOduration=1.519694052 podStartE2EDuration="1.519694052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:46.518366029 +0000 UTC m=+727.819322609" watchObservedRunningTime="2026-10-31 01:16:46.519694052 +0000 UTC m=+727.820650630"
Oct 31 01:16:46.665324 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:46.665222    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" podStartSLOduration=1.665208276 podStartE2EDuration="1.665208276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:46.663371313 +0000 UTC m=+727.964327908" watchObservedRunningTime="2026-10-31 01:16:46.665208276 +0000 UTC m=+727.966164876"
Oct 31 01:16:47.401995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.401959    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 31 01:16:47.402202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.402014    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 31 01:16:47.666061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.664243    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:58030->10.133.0.53:15021: read: connection reset by peer" start-of-body=
Oct 31 01:16:47.666061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.664312    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:58030->10.133.0.53:15021: read: connection reset by peer"
Oct 31 01:16:47.768969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.768935    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"]
Oct 31 01:16:47.772404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.772387    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.814693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.814664    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbvb\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.814870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.814700    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.814870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.814739    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.814966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.814885    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.814966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.814913    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.814966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.814940    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.815297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.815029    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.815297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.815064    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.815297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.815083    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.824441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.824414    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057e2d-openshift-default-dockercfg-2bvxk\""
Oct 31 01:16:47.849118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.849052    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"]
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916229    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916279    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916335    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbvb\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916366    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916399    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916482    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916518    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916546    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916588    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.916980    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.917241    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.917337    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.917558    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.918164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.918009    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.920540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.920514    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.920812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.920791    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.974030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.973996    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbvb\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:47.974241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:47.974100    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:48.104921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.104889    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:48.405341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.405309    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:16:48.405618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.405598    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:16:48.406446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.406431    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:16:48.416584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.416558    5419 generic.go:356] "Generic (PLEG): container finished" podID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerID="c2a9da38006f89c7e4d3e7bc8f0803ab6ab38dae1d395d3776d5f8255ee3a02f" exitCode=0
Oct 31 01:16:48.416687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.416625    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" event={"ID":"cb3bb1ca-6e4a-465f-bb37-18141137d234","Type":"ContainerDied","Data":"c2a9da38006f89c7e4d3e7bc8f0803ab6ab38dae1d395d3776d5f8255ee3a02f"}
Oct 31 01:16:48.417106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.417093    5419 scope.go:122] "RemoveContainer" containerID="c2a9da38006f89c7e4d3e7bc8f0803ab6ab38dae1d395d3776d5f8255ee3a02f"
Oct 31 01:16:48.418374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.418347    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:48.418475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.418410    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:48.418475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.418447    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:48.573599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.573568    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"]
Oct 31 01:16:48.576956 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:48.576931    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e787433_9dc5_4e38_912f_b14630d9dda4.slice/crio-a80a42253ab3220fe4760a219c5c6023b90b075000dac5de8b4b68e6a8d91ad2 WatchSource:0}: Error finding container a80a42253ab3220fe4760a219c5c6023b90b075000dac5de8b4b68e6a8d91ad2: Status 404 returned error can't find the container with id a80a42253ab3220fe4760a219c5c6023b90b075000dac5de8b4b68e6a8d91ad2
Oct 31 01:16:48.579257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.579232    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:48.579342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.579290    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:48.579342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:48.579320    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:16:49.423219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:49.423180    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" event={"ID":"cb3bb1ca-6e4a-465f-bb37-18141137d234","Type":"ContainerStarted","Data":"811463d03de9539ef2b80a46f03f469be89517a71d956c0232beecc4e3d4f691"}
Oct 31 01:16:49.424912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:49.424885    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" event={"ID":"4e787433-9dc5-4e38-912f-b14630d9dda4","Type":"ContainerStarted","Data":"134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"}
Oct 31 01:16:49.425067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:49.424947    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" event={"ID":"4e787433-9dc5-4e38-912f-b14630d9dda4","Type":"ContainerStarted","Data":"a80a42253ab3220fe4760a219c5c6023b90b075000dac5de8b4b68e6a8d91ad2"}
Oct 31 01:16:49.669269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:49.669217    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podStartSLOduration=2.669202888 podStartE2EDuration="2.669202888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:49.667613823 +0000 UTC m=+730.968570391" watchObservedRunningTime="2026-10-31 01:16:49.669202888 +0000 UTC m=+730.970159465"
Oct 31 01:16:50.022858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.022827    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:50.035271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.035247    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:50.105929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.105896    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:50.110712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.110693    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:50.366017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.365924    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:50.366318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.366169    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:16:50.366318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.366232    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 01:16:50.430812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.430781    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:50.431684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.431652    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:16:50.441960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:50.441935    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:16:51.366104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:51.366062    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:16:51.366314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:51.366119    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 01:16:52.366309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:52.366272    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:16:52.366713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:52.366328    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 01:16:52.811486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:52.811448    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:16:53.354485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:53.354459    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 01:16:53.354686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:53.354524    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 01:16:53.366341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:53.366312    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:16:53.366707 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:53.366361    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 01:16:54.366106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:54.366068    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 31 01:16:54.366305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:54.366133    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 31 01:16:55.370354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:55.370322    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:55.370731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:55.370651    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:55.371379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:55.371357    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 01:16:56.967534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:56.967501    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"]
Oct 31 01:16:56.971091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:56.971073    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.022744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.022710    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-057e2d\""
Oct 31 01:16:57.030249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.030220    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgrm\" (UniqueName: \"kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm\") pod \"payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.030402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.030281    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.072167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.072121    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-057e2d-dockercfg-fbh57\""
Oct 31 01:16:57.074801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.074723    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"]
Oct 31 01:16:57.117115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.117078    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"]
Oct 31 01:16:57.120593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.120576    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.131053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.131030    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgrm\" (UniqueName: \"kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm\") pod \"payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.131195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.131082    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.131913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.131894    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.173698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.173670    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"]
Oct 31 01:16:57.188567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.188532    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgrm\" (UniqueName: \"kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm\") pod \"payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.232760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.232727    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l55z\" (UniqueName: \"kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z\") pod \"payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.232893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.232793    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.287226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.287188    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:57.334106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.334059    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l55z\" (UniqueName: \"kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z\") pod \"payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.334106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.334102    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.334959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.334915    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.392663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.392623    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l55z\" (UniqueName: \"kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z\") pod \"payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.414013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.413986    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 01:16:57.414242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.414058    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 01:16:57.434890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.434853    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:57.658287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.658261    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"]
Oct 31 01:16:57.661194 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:57.661165    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9019937e_21f8_49d3_b2e3_fbfe90eaffdf.slice/crio-47df04785de17794b5aa7599caae355b7a334b69175c15ef6e26c56fb3bda9ce WatchSource:0}: Error finding container 47df04785de17794b5aa7599caae355b7a334b69175c15ef6e26c56fb3bda9ce: Status 404 returned error can't find the container with id 47df04785de17794b5aa7599caae355b7a334b69175c15ef6e26c56fb3bda9ce
Oct 31 01:16:57.807599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:57.807524    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"]
Oct 31 01:16:57.810481 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:16:57.810450    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b3b334_2c23_4df0_9410_c4695698a5fd.slice/crio-3da5f62dbfe0b559aafb2b0fc6f2d6eee67973edff1d36a53bb5b00616a6bc6e WatchSource:0}: Error finding container 3da5f62dbfe0b559aafb2b0fc6f2d6eee67973edff1d36a53bb5b00616a6bc6e: Status 404 returned error can't find the container with id 3da5f62dbfe0b559aafb2b0fc6f2d6eee67973edff1d36a53bb5b00616a6bc6e
Oct 31 01:16:58.472079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.472043    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" event={"ID":"43b3b334-2c23-4df0-9410-c4695698a5fd","Type":"ContainerStarted","Data":"9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688"}
Oct 31 01:16:58.472614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.472104    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" event={"ID":"43b3b334-2c23-4df0-9410-c4695698a5fd","Type":"ContainerStarted","Data":"3da5f62dbfe0b559aafb2b0fc6f2d6eee67973edff1d36a53bb5b00616a6bc6e"}
Oct 31 01:16:58.472614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.472241    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:16:58.473571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.473543    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" event={"ID":"9019937e-21f8-49d3-b2e3-fbfe90eaffdf","Type":"ContainerStarted","Data":"e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f"}
Oct 31 01:16:58.473681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.473590    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" event={"ID":"9019937e-21f8-49d3-b2e3-fbfe90eaffdf","Type":"ContainerStarted","Data":"47df04785de17794b5aa7599caae355b7a334b69175c15ef6e26c56fb3bda9ce"}
Oct 31 01:16:58.473681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.473619    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:16:58.586635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.586584    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" podStartSLOduration=1.586568397 podStartE2EDuration="1.586568397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:58.585112737 +0000 UTC m=+739.886069339" watchObservedRunningTime="2026-10-31 01:16:58.586568397 +0000 UTC m=+739.887524974"
Oct 31 01:16:58.700527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:16:58.699542    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" podStartSLOduration=2.699524378 podStartE2EDuration="2.699524378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:58.699028623 +0000 UTC m=+739.999985201" watchObservedRunningTime="2026-10-31 01:16:58.699524378 +0000 UTC m=+740.000480956"
Oct 31 01:17:06.852232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:06.852187    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:17:09.487612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:09.487581    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 01:17:09.488088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:09.487771    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 01:17:14.581387 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:17:14.581345    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:17:14.836265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:14.836193    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:17:31.825837 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:31.825794    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:17:32.659376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:32.659295    5419 generic.go:356] "Generic (PLEG): container finished" podID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerID="134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054" exitCode=0
Oct 31 01:17:32.659376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:32.659343    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" event={"ID":"4e787433-9dc5-4e38-912f-b14630d9dda4","Type":"ContainerDied","Data":"134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"}
Oct 31 01:17:32.659772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:32.659758    5419 scope.go:122] "RemoveContainer" containerID="134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"
Oct 31 01:17:32.661121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:32.661095    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:17:32.661197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:32.661179    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:17:32.661233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:32.661212    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:17:33.665994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:33.665959    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" event={"ID":"4e787433-9dc5-4e38-912f-b14630d9dda4","Type":"ContainerStarted","Data":"3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649"}
Oct 31 01:17:34.105391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:34.105361    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:17:34.106549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:34.106529    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:17:34.106611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:34.106576    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:17:35.105585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:35.105552    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:17:35.106002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:35.105608    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:17:35.431649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:35.431558    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:17:36.106063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:36.106030    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:17:36.106544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:36.106083    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:17:37.105963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.105931    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:17:37.106201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.105989    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:17:37.599300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.599258    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"]
Oct 31 01:17:37.603246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.603227    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.656724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.656690    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"openshift-service-ca.crt\""
Oct 31 01:17:37.656724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.656714    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"]
Oct 31 01:17:37.659253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.659228    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.659386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.659265    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.659447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.659418    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.659494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.659445    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64cx9\" (UniqueName: \"kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.659536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.659498    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.659536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.659529    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.703945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.703921    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"kube-root-ca.crt\""
Oct 31 01:17:37.704131 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.704091    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"simulator-w2-00e840-kserve-self-signed-certs\""
Oct 31 01:17:37.704216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.704194    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"default-dockercfg-bx7rh\""
Oct 31 01:17:37.760683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.760642    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.760683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.760684    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.760932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.760761    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.760932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.760785    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64cx9\" (UniqueName: \"kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.760932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.760853    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.760932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.760897    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.761096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.761039    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.761178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.761127    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.761335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.761305    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.763502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.763470    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.763631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.763611    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.818388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.818356    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64cx9\" (UniqueName: \"kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9\") pod \"simulator-w2-00e840-kserve-65466f45d-grg2w\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") " pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:37.915209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:37.915116    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:38.105812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.105775    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 01:17:38.105985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.105832    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 01:17:38.300551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.298477    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"]
Oct 31 01:17:38.301673 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:17:38.301645    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd0c6e95_4f05_4c05_9bcb_f5c82d05358f.slice/crio-1cb5d2c4a3d681b7da972354c04b622c53ef35ea2a47bd9708f0a5af9a9be133 WatchSource:0}: Error finding container 1cb5d2c4a3d681b7da972354c04b622c53ef35ea2a47bd9708f0a5af9a9be133: Status 404 returned error can't find the container with id 1cb5d2c4a3d681b7da972354c04b622c53ef35ea2a47bd9708f0a5af9a9be133
Oct 31 01:17:38.715770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.715687    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" event={"ID":"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f","Type":"ContainerStarted","Data":"37d99237226346e21a412dcd1979efd7cc39eae3f70d134a550d289cd0c5af5c"}
Oct 31 01:17:38.715770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.715737    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" event={"ID":"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f","Type":"ContainerStarted","Data":"1cb5d2c4a3d681b7da972354c04b622c53ef35ea2a47bd9708f0a5af9a9be133"}
Oct 31 01:17:38.835168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.834645    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podStartSLOduration=1.834625945 podStartE2EDuration="1.834625945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:38.834348558 +0000 UTC m=+780.135305135" watchObservedRunningTime="2026-10-31 01:17:38.834625945 +0000 UTC m=+780.135582524"
Oct 31 01:17:38.915312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:38.915275    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:39.110614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:39.110571    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:17:39.112756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:39.112726    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:17:44.581501 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:17:44.581455    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:17:46.313381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:46.313347    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:17:47.915543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:47.915509    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:47.928110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:47.928090    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:17:48.777293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:17:48.777263    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:18:08.401063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:08.401029    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:18:11.485935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.485903    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"]
Oct 31 01:18:11.488809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.488787    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.510584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.510553    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.510719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.510605    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snf2c\" (UniqueName: \"kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.510769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.510710    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.510846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.510819    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.510895 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.510875    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.510934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.510925    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.589865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.589840    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"openshift-service-ca.crt\""
Oct 31 01:18:11.590015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.589887    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"simulator-w4-8f6b5b-kserve-self-signed-certs\""
Oct 31 01:18:11.590015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.589922    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"default-dockercfg-28mfd\""
Oct 31 01:18:11.590015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.589963    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"kube-root-ca.crt\""
Oct 31 01:18:11.591844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.591825    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"]
Oct 31 01:18:11.612377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.612354    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.612492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.612388    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snf2c\" (UniqueName: \"kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.612492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.612411    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.612492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.612454    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.612492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.612483    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.612699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.612523    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.613062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.613032    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.613062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.613045    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.613292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.613273    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.615085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.615065    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.615403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.615381    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.669012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.668981    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snf2c\" (UniqueName: \"kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c\") pod \"simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") " pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:11.822227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:11.822127    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:12.195503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:12.195474    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"]
Oct 31 01:18:12.197642 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:18:12.197621    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef5ecb7d_467c_469a_8c66_d79d6149b570.slice/crio-cd7d7390234359b93142c2c45f599665888f45ba65cb4fdf527e4ee195a074c9 WatchSource:0}: Error finding container cd7d7390234359b93142c2c45f599665888f45ba65cb4fdf527e4ee195a074c9: Status 404 returned error can't find the container with id cd7d7390234359b93142c2c45f599665888f45ba65cb4fdf527e4ee195a074c9
Oct 31 01:18:12.895638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:12.895596    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" event={"ID":"ef5ecb7d-467c-469a-8c66-d79d6149b570","Type":"ContainerStarted","Data":"3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f"}
Oct 31 01:18:12.896036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:12.895651    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" event={"ID":"ef5ecb7d-467c-469a-8c66-d79d6149b570","Type":"ContainerStarted","Data":"cd7d7390234359b93142c2c45f599665888f45ba65cb4fdf527e4ee195a074c9"}
Oct 31 01:18:13.014196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:13.014101    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podStartSLOduration=2.014082385 podStartE2EDuration="2.014082385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:13.011430858 +0000 UTC m=+814.312387437" watchObservedRunningTime="2026-10-31 01:18:13.014082385 +0000 UTC m=+814.315038964"
Oct 31 01:18:13.823022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:13.822988    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:14.581079 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:18:14.581041    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:18:21.822869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:21.822782    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:21.835470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:21.835442    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:21.955462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:21.955435    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:18:25.991027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:25.990993    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:18:38.369995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.369959    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"]
Oct 31 01:18:38.373731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.373714    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.396378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.396350    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.396532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.396418    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnwz\" (UniqueName: \"kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.396575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.396548    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.430672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.430644    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"]
Oct 31 01:18:38.478042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.478009    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:18:38.478042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.478008    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:18:38.478258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.478095    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:18:38.478258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.478109    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:18:38.497652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.497625    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.497759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.497659    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.497759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.497715    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnwz\" (UniqueName: \"kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.498378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.498358    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.500120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.500101    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.554018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.553991    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnwz\" (UniqueName: \"kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz\") pod \"maas-api-revoke-sub-1483c614-2mplw\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:38.687246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:38.687184    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:39.059652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:39.059630    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"]
Oct 31 01:18:39.061950 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:18:39.061918    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34375ce_4ccf_4ba4_88be_5ca1b1b7fb06.slice/crio-a5317febccd60bec230511b6a14ad2fa5e5401905faebcc4538a9481345ea743 WatchSource:0}: Error finding container a5317febccd60bec230511b6a14ad2fa5e5401905faebcc4538a9481345ea743: Status 404 returned error can't find the container with id a5317febccd60bec230511b6a14ad2fa5e5401905faebcc4538a9481345ea743
Oct 31 01:18:40.044333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:40.044304    5419 generic.go:356] "Generic (PLEG): container finished" podID="c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" containerID="038ab5a0181933b037a98fe52db622d37dadd0c637a3aac3c6eb1812e0f2bd1f" exitCode=0
Oct 31 01:18:40.044697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:40.044385    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw" event={"ID":"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06","Type":"ContainerDied","Data":"038ab5a0181933b037a98fe52db622d37dadd0c637a3aac3c6eb1812e0f2bd1f"}
Oct 31 01:18:40.044697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:40.044435    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw" event={"ID":"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06","Type":"ContainerStarted","Data":"a5317febccd60bec230511b6a14ad2fa5e5401905faebcc4538a9481345ea743"}
Oct 31 01:18:41.193752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.193727    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:41.223505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.223483    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz\" (UniqueName: \"kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz\") pod \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") "
Oct 31 01:18:41.223619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.223519    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca\") pod \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") "
Oct 31 01:18:41.223619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.223546    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca\") pod \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\" (UID: \"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06\") "
Oct 31 01:18:41.223919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.223894    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca" pod "c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" (UID: "c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:41.225698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.225672    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca" pod "c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" (UID: "c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:18:41.225798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.225742    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz" pod "c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" (UID: "c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06"). InnerVolumeSpecName "kube-api-access-zwnwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:41.324824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.324751    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnwz\" (UniqueName: \"kubernetes.io/projected/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-kube-api-access-zwnwz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:41.324824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.324780    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:41.324824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:41.324791    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:42.056885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:42.056845    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw" event={"ID":"c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06","Type":"ContainerDied","Data":"a5317febccd60bec230511b6a14ad2fa5e5401905faebcc4538a9481345ea743"}
Oct 31 01:18:42.056885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:42.056863    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"
Oct 31 01:18:42.057099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:42.056896    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5317febccd60bec230511b6a14ad2fa5e5401905faebcc4538a9481345ea743"
Oct 31 01:18:44.581762 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:18:44.581724    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:18:56.104432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.104399    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"]
Oct 31 01:18:56.104852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.104819    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" containerName="revoke-keys"
Oct 31 01:18:56.104852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.104830    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" containerName="revoke-keys"
Oct 31 01:18:56.104940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.104928    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" containerName="revoke-keys"
Oct 31 01:18:56.107949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.107934    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.164881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.164842    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"]
Oct 31 01:18:56.173765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.173732    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.173927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.173852    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xts72\" (UniqueName: \"kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.173927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.173919    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.212002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.211976    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:18:56.212002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.211990    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:18:56.212002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.211996    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:18:56.212256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.212033    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:18:56.274924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.274892    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xts72\" (UniqueName: \"kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.275103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.274950    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.275103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.275009    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.275593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.275573    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.277489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.277465    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.331867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.331841    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xts72\" (UniqueName: \"kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72\") pod \"maas-api-revoke-sub-052d2b3f-mmrz7\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.420828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.420746    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:56.799239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:56.799215    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"]
Oct 31 01:18:56.801902 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:18:56.801876    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33362e87_d320_46ac_8cc2_2e8d767e1f3a.slice/crio-365d31a4b14646a775bb5b03f264e83788d1110ea2104bb380f69752cc4c63d2 WatchSource:0}: Error finding container 365d31a4b14646a775bb5b03f264e83788d1110ea2104bb380f69752cc4c63d2: Status 404 returned error can't find the container with id 365d31a4b14646a775bb5b03f264e83788d1110ea2104bb380f69752cc4c63d2
Oct 31 01:18:57.094595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:57.094569    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:18:57.138085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:57.138047    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7" event={"ID":"33362e87-d320-46ac-8cc2-2e8d767e1f3a","Type":"ContainerStarted","Data":"4765449f254ac63977503c34a3300da4750e1e0b405266f8d3412facf090d785"}
Oct 31 01:18:57.138490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:57.138108    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7" event={"ID":"33362e87-d320-46ac-8cc2-2e8d767e1f3a","Type":"ContainerStarted","Data":"365d31a4b14646a775bb5b03f264e83788d1110ea2104bb380f69752cc4c63d2"}
Oct 31 01:18:57.252889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:57.252815    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7" podStartSLOduration=1.252795124 podStartE2EDuration="1.252795124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:57.252644677 +0000 UTC m=+858.553601255" watchObservedRunningTime="2026-10-31 01:18:57.252795124 +0000 UTC m=+858.553751702"
Oct 31 01:18:58.144719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:58.144687    5419 generic.go:356] "Generic (PLEG): container finished" podID="33362e87-d320-46ac-8cc2-2e8d767e1f3a" containerID="4765449f254ac63977503c34a3300da4750e1e0b405266f8d3412facf090d785" exitCode=0
Oct 31 01:18:58.145113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:58.144750    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7" event={"ID":"33362e87-d320-46ac-8cc2-2e8d767e1f3a","Type":"ContainerDied","Data":"4765449f254ac63977503c34a3300da4750e1e0b405266f8d3412facf090d785"}
Oct 31 01:18:59.286399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.286373    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:18:59.406897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.406820    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca\") pod \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") "
Oct 31 01:18:59.407064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.406914    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca\") pod \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") "
Oct 31 01:18:59.407064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.407010    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72\" (UniqueName: \"kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72\") pod \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\" (UID: \"33362e87-d320-46ac-8cc2-2e8d767e1f3a\") "
Oct 31 01:18:59.407273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.407236    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca" pod "33362e87-d320-46ac-8cc2-2e8d767e1f3a" (UID: "33362e87-d320-46ac-8cc2-2e8d767e1f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:18:59.407401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.407367    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:59.409305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.409275    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72" pod "33362e87-d320-46ac-8cc2-2e8d767e1f3a" (UID: "33362e87-d320-46ac-8cc2-2e8d767e1f3a"). InnerVolumeSpecName "kube-api-access-xts72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:18:59.409414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.409364    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca" pod "33362e87-d320-46ac-8cc2-2e8d767e1f3a" (UID: "33362e87-d320-46ac-8cc2-2e8d767e1f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:18:59.508820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.508789    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xts72\" (UniqueName: \"kubernetes.io/projected/33362e87-d320-46ac-8cc2-2e8d767e1f3a-kube-api-access-xts72\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:18:59.508820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:18:59.508818    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33362e87-d320-46ac-8cc2-2e8d767e1f3a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:00.155857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:00.155815    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7" event={"ID":"33362e87-d320-46ac-8cc2-2e8d767e1f3a","Type":"ContainerDied","Data":"365d31a4b14646a775bb5b03f264e83788d1110ea2104bb380f69752cc4c63d2"}
Oct 31 01:19:00.155857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:00.155840    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"
Oct 31 01:19:00.155857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:00.155863    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="365d31a4b14646a775bb5b03f264e83788d1110ea2104bb380f69752cc4c63d2"
Oct 31 01:19:07.470020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.469987    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"]
Oct 31 01:19:07.470432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.470404    5419 cpu_manager.go:414] "containerMap: removing container" podUID="33362e87-d320-46ac-8cc2-2e8d767e1f3a" containerName="revoke-keys"
Oct 31 01:19:07.470432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.470415    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33362e87-d320-46ac-8cc2-2e8d767e1f3a" containerName="revoke-keys"
Oct 31 01:19:07.470509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.470499    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="33362e87-d320-46ac-8cc2-2e8d767e1f3a" containerName="revoke-keys"
Oct 31 01:19:07.473743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.473727    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.518247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.518211    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddx6\" (UniqueName: \"kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.518399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.518268    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.518399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.518378    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.518482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.518405    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.518482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.518436    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.518597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.518572    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.575046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.575010    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"premium-w4-8f6b5b-kserve-self-signed-certs\""
Oct 31 01:19:07.576995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.576973    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"]
Oct 31 01:19:07.619644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.619613    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.619794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.619651    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.619794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.619695    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddx6\" (UniqueName: \"kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.619794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.619741    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.619941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.619812    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.619941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.619842    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.620187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.620159    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.620301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.620211    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.620301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.620241    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.622373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.622347    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.622583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.622561    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.677171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.677106    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddx6\" (UniqueName: \"kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6\") pod \"premium-w4-8f6b5b-kserve-648dc97bb5-trpxj\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") " pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:07.791918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:07.791839    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:08.188511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:08.188314    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"]
Oct 31 01:19:08.191299 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:19:08.191271    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3041fe71_9784_4551_a2a4_4d48a4a10ae0.slice/crio-32304d958f6ace6a96dbb5904a804a6a4a0159580027e1d78fbb62ade243683f WatchSource:0}: Error finding container 32304d958f6ace6a96dbb5904a804a6a4a0159580027e1d78fbb62ade243683f: Status 404 returned error can't find the container with id 32304d958f6ace6a96dbb5904a804a6a4a0159580027e1d78fbb62ade243683f
Oct 31 01:19:08.202213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:08.202180    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" event={"ID":"3041fe71-9784-4551-a2a4-4d48a4a10ae0","Type":"ContainerStarted","Data":"32304d958f6ace6a96dbb5904a804a6a4a0159580027e1d78fbb62ade243683f"}
Oct 31 01:19:09.209114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:09.209059    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" event={"ID":"3041fe71-9784-4551-a2a4-4d48a4a10ae0","Type":"ContainerStarted","Data":"c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81"}
Oct 31 01:19:09.323827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:09.323774    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podStartSLOduration=2.323760344 podStartE2EDuration="2.323760344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:09.322750917 +0000 UTC m=+870.623707488" watchObservedRunningTime="2026-10-31 01:19:09.323760344 +0000 UTC m=+870.624716936"
Oct 31 01:19:09.792683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:09.792655    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:14.580849 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:19:14.580811    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:19:15.885931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:15.885896    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:19:17.792430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:17.792392    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:17.804861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:17.804838    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:18.270980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:18.270945    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:19:28.309759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.309715    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"]
Oct 31 01:19:28.313936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.313914    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.370452 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.370424    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"]
Oct 31 01:19:28.417896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.417867    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:19:28.418062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.417923    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:19:28.418062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.417919    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:19:28.418062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.418029    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:19:28.435903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.435873    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ms7\" (UniqueName: \"kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.436020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.435949    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.436089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.436069    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.536734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.536701    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.536912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.536764    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.536912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.536854    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ms7\" (UniqueName: \"kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.537406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.537381    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.539258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.539234    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.593589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.593530    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ms7\" (UniqueName: \"kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7\") pod \"maas-api-revoke-sub-019af69e-djl76\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.627754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.627711    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:28.997634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:28.997612    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"]
Oct 31 01:19:28.999752 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:19:28.999704    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e64e59_96bd_426c_a43b_ce25a2666532.slice/crio-a0c629391573cc4362b9e90717c730b50e70f2d24b68b23ce07bfae7ef979a6d WatchSource:0}: Error finding container a0c629391573cc4362b9e90717c730b50e70f2d24b68b23ce07bfae7ef979a6d: Status 404 returned error can't find the container with id a0c629391573cc4362b9e90717c730b50e70f2d24b68b23ce07bfae7ef979a6d
Oct 31 01:19:29.321966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:29.321930    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76" event={"ID":"c5e64e59-96bd-426c-a43b-ce25a2666532","Type":"ContainerStarted","Data":"ee044be81e42180181dffab872dbc6e6631a0b19dfe2cd6ea1a7ed691b741bf1"}
Oct 31 01:19:29.322363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:29.321987    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76" event={"ID":"c5e64e59-96bd-426c-a43b-ce25a2666532","Type":"ContainerStarted","Data":"a0c629391573cc4362b9e90717c730b50e70f2d24b68b23ce07bfae7ef979a6d"}
Oct 31 01:19:29.435918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:29.435854    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76" podStartSLOduration=1.435833544 podStartE2EDuration="1.435833544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:29.434919707 +0000 UTC m=+890.735876284" watchObservedRunningTime="2026-10-31 01:19:29.435833544 +0000 UTC m=+890.736790122"
Oct 31 01:19:30.329533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:30.329499    5419 generic.go:356] "Generic (PLEG): container finished" podID="c5e64e59-96bd-426c-a43b-ce25a2666532" containerID="ee044be81e42180181dffab872dbc6e6631a0b19dfe2cd6ea1a7ed691b741bf1" exitCode=0
Oct 31 01:19:30.329943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:30.329563    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76" event={"ID":"c5e64e59-96bd-426c-a43b-ce25a2666532","Type":"ContainerDied","Data":"ee044be81e42180181dffab872dbc6e6631a0b19dfe2cd6ea1a7ed691b741bf1"}
Oct 31 01:19:31.474020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.473995    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:31.566934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.566905    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca\") pod \"c5e64e59-96bd-426c-a43b-ce25a2666532\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") "
Oct 31 01:19:31.567124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.566959    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca\") pod \"c5e64e59-96bd-426c-a43b-ce25a2666532\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") "
Oct 31 01:19:31.567124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.567040    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7\" (UniqueName: \"kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7\") pod \"c5e64e59-96bd-426c-a43b-ce25a2666532\" (UID: \"c5e64e59-96bd-426c-a43b-ce25a2666532\") "
Oct 31 01:19:31.567356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.567334    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca" pod "c5e64e59-96bd-426c-a43b-ce25a2666532" (UID: "c5e64e59-96bd-426c-a43b-ce25a2666532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:31.569191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.569169    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca" pod "c5e64e59-96bd-426c-a43b-ce25a2666532" (UID: "c5e64e59-96bd-426c-a43b-ce25a2666532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:31.569298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.569278    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7" pod "c5e64e59-96bd-426c-a43b-ce25a2666532" (UID: "c5e64e59-96bd-426c-a43b-ce25a2666532"). InnerVolumeSpecName "kube-api-access-l2ms7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:31.668111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.668035    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:31.668111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.668065    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e64e59-96bd-426c-a43b-ce25a2666532-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:31.668111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:31.668078    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ms7\" (UniqueName: \"kubernetes.io/projected/c5e64e59-96bd-426c-a43b-ce25a2666532-kube-api-access-l2ms7\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:32.341204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:32.341170    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"
Oct 31 01:19:32.341204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:32.341190    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76" event={"ID":"c5e64e59-96bd-426c-a43b-ce25a2666532","Type":"ContainerDied","Data":"a0c629391573cc4362b9e90717c730b50e70f2d24b68b23ce07bfae7ef979a6d"}
Oct 31 01:19:32.341418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:32.341239    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c629391573cc4362b9e90717c730b50e70f2d24b68b23ce07bfae7ef979a6d"
Oct 31 01:19:34.495767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:34.495735    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:19:37.061842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.061806    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"]
Oct 31 01:19:37.062438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.062419    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c5e64e59-96bd-426c-a43b-ce25a2666532" containerName="revoke-keys"
Oct 31 01:19:37.062526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.062441    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e64e59-96bd-426c-a43b-ce25a2666532" containerName="revoke-keys"
Oct 31 01:19:37.062608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.062595    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e64e59-96bd-426c-a43b-ce25a2666532" containerName="revoke-keys"
Oct 31 01:19:37.065946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.065925    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.122261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.122233    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"]
Oct 31 01:19:37.169338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.169305    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:19:37.169504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.169305    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:19:37.169504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.169306    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:19:37.169504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.169370    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:19:37.224564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.224530    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kpkm\" (UniqueName: \"kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.224564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.224565    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.224759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.224675    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.326048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.325962    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.326261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.326049    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kpkm\" (UniqueName: \"kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.326261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.326074    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.326725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.326705    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.328622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.328594    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.383009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.382973    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kpkm\" (UniqueName: \"kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm\") pod \"maas-api-revoke-sub-c8c3125a-4sfc8\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:37.679371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:37.679290    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:38.048676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:38.048651    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"]
Oct 31 01:19:38.051006 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:19:38.050979    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90219e1d_896a_45c8_adca_9b54caf26764.slice/crio-b3d8387c310542da886a31e5507a5d5a9656ba314ff6c2cd844757a8262b5882 WatchSource:0}: Error finding container b3d8387c310542da886a31e5507a5d5a9656ba314ff6c2cd844757a8262b5882: Status 404 returned error can't find the container with id b3d8387c310542da886a31e5507a5d5a9656ba314ff6c2cd844757a8262b5882
Oct 31 01:19:38.375703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:38.375608    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8" event={"ID":"90219e1d-896a-45c8-adca-9b54caf26764","Type":"ContainerStarted","Data":"b3d8387c310542da886a31e5507a5d5a9656ba314ff6c2cd844757a8262b5882"}
Oct 31 01:19:39.382169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:39.382115    5419 generic.go:356] "Generic (PLEG): container finished" podID="90219e1d-896a-45c8-adca-9b54caf26764" containerID="4d2d8a77c53fead96a69aca01b62a45869f62c650beff8e1975319bcd5af45f4" exitCode=0
Oct 31 01:19:39.382561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:39.382201    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8" event={"ID":"90219e1d-896a-45c8-adca-9b54caf26764","Type":"ContainerDied","Data":"4d2d8a77c53fead96a69aca01b62a45869f62c650beff8e1975319bcd5af45f4"}
Oct 31 01:19:40.525337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.525310    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:40.658261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.658175    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca\") pod \"90219e1d-896a-45c8-adca-9b54caf26764\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") "
Oct 31 01:19:40.658261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.658221    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm\" (UniqueName: \"kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm\") pod \"90219e1d-896a-45c8-adca-9b54caf26764\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") "
Oct 31 01:19:40.658616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.658268    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca\") pod \"90219e1d-896a-45c8-adca-9b54caf26764\" (UID: \"90219e1d-896a-45c8-adca-9b54caf26764\") "
Oct 31 01:19:40.658714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.658691    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca" pod "90219e1d-896a-45c8-adca-9b54caf26764" (UID: "90219e1d-896a-45c8-adca-9b54caf26764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:40.660565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.660534    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm" pod "90219e1d-896a-45c8-adca-9b54caf26764" (UID: "90219e1d-896a-45c8-adca-9b54caf26764"). InnerVolumeSpecName "kube-api-access-9kpkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:40.660708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.660685    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca" pod "90219e1d-896a-45c8-adca-9b54caf26764" (UID: "90219e1d-896a-45c8-adca-9b54caf26764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:40.759561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.759526    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:40.759561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.759555    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kpkm\" (UniqueName: \"kubernetes.io/projected/90219e1d-896a-45c8-adca-9b54caf26764-kube-api-access-9kpkm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:40.759561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:40.759565    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90219e1d-896a-45c8-adca-9b54caf26764-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:41.394386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:41.394358    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"
Oct 31 01:19:41.394562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:41.394381    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8" event={"ID":"90219e1d-896a-45c8-adca-9b54caf26764","Type":"ContainerDied","Data":"b3d8387c310542da886a31e5507a5d5a9656ba314ff6c2cd844757a8262b5882"}
Oct 31 01:19:41.394562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:41.394432    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d8387c310542da886a31e5507a5d5a9656ba314ff6c2cd844757a8262b5882"
Oct 31 01:19:44.190528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.190489    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"]
Oct 31 01:19:44.191266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.191243    5419 cpu_manager.go:414] "containerMap: removing container" podUID="90219e1d-896a-45c8-adca-9b54caf26764" containerName="revoke-keys"
Oct 31 01:19:44.191266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.191267    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90219e1d-896a-45c8-adca-9b54caf26764" containerName="revoke-keys"
Oct 31 01:19:44.191479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.191465    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="90219e1d-896a-45c8-adca-9b54caf26764" containerName="revoke-keys"
Oct 31 01:19:44.195764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.195742    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.251955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.251843    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"]
Oct 31 01:19:44.297575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.297516    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:19:44.297575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.297540    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:19:44.297575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.297516    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:19:44.297836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.297732    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:19:44.299456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.299431    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.299650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.299628    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.299720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.299674    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k6qx\" (UniqueName: \"kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.401204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.401172    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.401405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.401219    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k6qx\" (UniqueName: \"kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.401405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.401282    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.402061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.402031    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.403752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.403725    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.458531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.458462    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k6qx\" (UniqueName: \"kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx\") pod \"maas-api-revoke-sub-5089baf5-g2g67\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.515656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.515624    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:44.611366 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:19:44.610807    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:19:44.905497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:44.905393    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"]
Oct 31 01:19:44.908437 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:19:44.908413    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4342de72_0f8c_4eea_b454_bd2aa9b82fb3.slice/crio-df41c22c32fda4cf77500dc22a562b7df6cf021a49a91f1335bd92a7812e6326 WatchSource:0}: Error finding container df41c22c32fda4cf77500dc22a562b7df6cf021a49a91f1335bd92a7812e6326: Status 404 returned error can't find the container with id df41c22c32fda4cf77500dc22a562b7df6cf021a49a91f1335bd92a7812e6326
Oct 31 01:19:45.419051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:45.419005    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67" event={"ID":"4342de72-0f8c-4eea-b454-bd2aa9b82fb3","Type":"ContainerStarted","Data":"f367c692a7af4a63819d0657bf48c5597b01cda33ad83a64b6d623ddb4a52ada"}
Oct 31 01:19:45.419461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:45.419072    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67" event={"ID":"4342de72-0f8c-4eea-b454-bd2aa9b82fb3","Type":"ContainerStarted","Data":"df41c22c32fda4cf77500dc22a562b7df6cf021a49a91f1335bd92a7812e6326"}
Oct 31 01:19:45.535578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:45.535524    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67" podStartSLOduration=1.535507248 podStartE2EDuration="1.535507248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:45.53329151 +0000 UTC m=+906.834248216" watchObservedRunningTime="2026-10-31 01:19:45.535507248 +0000 UTC m=+906.836463828"
Oct 31 01:19:46.427398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:46.427363    5419 generic.go:356] "Generic (PLEG): container finished" podID="4342de72-0f8c-4eea-b454-bd2aa9b82fb3" containerID="f367c692a7af4a63819d0657bf48c5597b01cda33ad83a64b6d623ddb4a52ada" exitCode=0
Oct 31 01:19:46.427805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:46.427444    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67" event={"ID":"4342de72-0f8c-4eea-b454-bd2aa9b82fb3","Type":"ContainerDied","Data":"f367c692a7af4a63819d0657bf48c5597b01cda33ad83a64b6d623ddb4a52ada"}
Oct 31 01:19:47.566638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.566603    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:47.737625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.737593    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx\" (UniqueName: \"kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx\") pod \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") "
Oct 31 01:19:47.737793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.737644    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca\") pod \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") "
Oct 31 01:19:47.737793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.737754    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca\") pod \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\" (UID: \"4342de72-0f8c-4eea-b454-bd2aa9b82fb3\") "
Oct 31 01:19:47.738083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.738062    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca" pod "4342de72-0f8c-4eea-b454-bd2aa9b82fb3" (UID: "4342de72-0f8c-4eea-b454-bd2aa9b82fb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:47.739920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.739896    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca" pod "4342de72-0f8c-4eea-b454-bd2aa9b82fb3" (UID: "4342de72-0f8c-4eea-b454-bd2aa9b82fb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:47.739977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.739939    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx" pod "4342de72-0f8c-4eea-b454-bd2aa9b82fb3" (UID: "4342de72-0f8c-4eea-b454-bd2aa9b82fb3"). InnerVolumeSpecName "kube-api-access-4k6qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:47.838689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.838657    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:47.838689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.838683    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k6qx\" (UniqueName: \"kubernetes.io/projected/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-kube-api-access-4k6qx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:47.838689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:47.838692    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4342de72-0f8c-4eea-b454-bd2aa9b82fb3-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:48.440299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:48.440265    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"
Oct 31 01:19:48.440299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:48.440289    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67" event={"ID":"4342de72-0f8c-4eea-b454-bd2aa9b82fb3","Type":"ContainerDied","Data":"df41c22c32fda4cf77500dc22a562b7df6cf021a49a91f1335bd92a7812e6326"}
Oct 31 01:19:48.440549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:48.440342    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df41c22c32fda4cf77500dc22a562b7df6cf021a49a91f1335bd92a7812e6326"
Oct 31 01:19:49.258387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.258352    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"]
Oct 31 01:19:49.258800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.258787    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4342de72-0f8c-4eea-b454-bd2aa9b82fb3" containerName="revoke-keys"
Oct 31 01:19:49.258864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.258801    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4342de72-0f8c-4eea-b454-bd2aa9b82fb3" containerName="revoke-keys"
Oct 31 01:19:49.258901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.258873    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4342de72-0f8c-4eea-b454-bd2aa9b82fb3" containerName="revoke-keys"
Oct 31 01:19:49.262058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.262036    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.318767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.318739    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"]
Oct 31 01:19:49.344744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.344709    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:19:49.344922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.344797    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:19:49.363913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.363875    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qr7\" (UniqueName: \"kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.364079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.364017    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.364184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.364104    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.365213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.365196    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:19:49.365301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.365250    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:19:49.464981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.464947    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.465184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.465064    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qr7\" (UniqueName: \"kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.465242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.465208    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.465831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.465809    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.467519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.467501    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.522213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.522124    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qr7\" (UniqueName: \"kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7\") pod \"maas-api-revoke-sub-e7bec8f7-wplwt\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.575210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.575182    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:49.948175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:49.948149    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"]
Oct 31 01:19:49.950888 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:19:49.950862    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cf3e7f_b2d2_4ef7_b4f2_2640ad9fce6a.slice/crio-51ed7ee72e67d61cb33fba43ac3dff3bbf9731a6e332367c8967a9092f7a8e74 WatchSource:0}: Error finding container 51ed7ee72e67d61cb33fba43ac3dff3bbf9731a6e332367c8967a9092f7a8e74: Status 404 returned error can't find the container with id 51ed7ee72e67d61cb33fba43ac3dff3bbf9731a6e332367c8967a9092f7a8e74
Oct 31 01:19:50.453377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:50.453341    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt" event={"ID":"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a","Type":"ContainerStarted","Data":"9694594dab2129ae8c5f3e64653f2637df22d63a18bfcde391a6923c03da0221"}
Oct 31 01:19:50.453807 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:50.453402    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt" event={"ID":"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a","Type":"ContainerStarted","Data":"51ed7ee72e67d61cb33fba43ac3dff3bbf9731a6e332367c8967a9092f7a8e74"}
Oct 31 01:19:50.569360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:50.569264    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt" podStartSLOduration=1.569249396 podStartE2EDuration="1.569249396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:50.566905051 +0000 UTC m=+911.867861643" watchObservedRunningTime="2026-10-31 01:19:50.569249396 +0000 UTC m=+911.870205972"
Oct 31 01:19:51.460434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:51.460401    5419 generic.go:356] "Generic (PLEG): container finished" podID="b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" containerID="9694594dab2129ae8c5f3e64653f2637df22d63a18bfcde391a6923c03da0221" exitCode=0
Oct 31 01:19:51.460879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:51.460486    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt" event={"ID":"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a","Type":"ContainerDied","Data":"9694594dab2129ae8c5f3e64653f2637df22d63a18bfcde391a6923c03da0221"}
Oct 31 01:19:52.601552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.601521    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:52.697015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.696989    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca\") pod \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") "
Oct 31 01:19:52.697191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.697057    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca\") pod \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") "
Oct 31 01:19:52.697191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.697118    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7\" (UniqueName: \"kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7\") pod \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\" (UID: \"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a\") "
Oct 31 01:19:52.697443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.697421    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca" pod "b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" (UID: "b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:19:52.697566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.697552    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:52.699348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.699325    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca" pod "b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" (UID: "b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:19:52.699348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.699330    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7" pod "b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" (UID: "b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a"). InnerVolumeSpecName "kube-api-access-c2qr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:19:52.798212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.798177    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:52.798212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:52.798205    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qr7\" (UniqueName: \"kubernetes.io/projected/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a-kube-api-access-c2qr7\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:19:53.473382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:53.473353    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"
Oct 31 01:19:53.473560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:53.473382    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt" event={"ID":"b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a","Type":"ContainerDied","Data":"51ed7ee72e67d61cb33fba43ac3dff3bbf9731a6e332367c8967a9092f7a8e74"}
Oct 31 01:19:53.473560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:53.473432    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51ed7ee72e67d61cb33fba43ac3dff3bbf9731a6e332367c8967a9092f7a8e74"
Oct 31 01:19:58.716927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:19:58.716884    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:20:05.247718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.247678    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"]
Oct 31 01:20:05.248115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.248106    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" containerName="revoke-keys"
Oct 31 01:20:05.248194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.248119    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" containerName="revoke-keys"
Oct 31 01:20:05.248236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.248231    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" containerName="revoke-keys"
Oct 31 01:20:05.251680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.251665    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.310602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.310562    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"]
Oct 31 01:20:05.330647 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.330615    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.330812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.330669    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjm2c\" (UniqueName: \"kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.330812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.330783    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.362033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.362001    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:20:05.362033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.362025    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:20:05.362263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.362048    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:20:05.362263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.362094    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:20:05.431883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.431850    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.432044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.431968    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.432044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.432028    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjm2c\" (UniqueName: \"kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.432636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.432617    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.434455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.434432    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.495115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.495079    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjm2c\" (UniqueName: \"kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c\") pod \"maas-api-revoke-sub-e1a85e88-pfsq6\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.568649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.568565    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:05.960849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:05.960828    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"]
Oct 31 01:20:05.962913 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:05.962880    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9daaaaf_047e_40b9_ab47_b01ad52e7e1f.slice/crio-892f3cd51c7da3502c71d85b6381d41739e808a6497d66fb248be91c21ad03d6 WatchSource:0}: Error finding container 892f3cd51c7da3502c71d85b6381d41739e808a6497d66fb248be91c21ad03d6: Status 404 returned error can't find the container with id 892f3cd51c7da3502c71d85b6381d41739e808a6497d66fb248be91c21ad03d6
Oct 31 01:20:06.545217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:06.545102    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6" event={"ID":"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f","Type":"ContainerStarted","Data":"6a18f86b99ec85b60cbea4b6df3081f37fd87e0996a098e20ad809b9b2674062"}
Oct 31 01:20:06.545217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:06.545177    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6" event={"ID":"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f","Type":"ContainerStarted","Data":"892f3cd51c7da3502c71d85b6381d41739e808a6497d66fb248be91c21ad03d6"}
Oct 31 01:20:06.659294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:06.659228    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6" podStartSLOduration=1.6592102020000001 podStartE2EDuration="1.659210202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:06.658817142 +0000 UTC m=+927.959773723" watchObservedRunningTime="2026-10-31 01:20:06.659210202 +0000 UTC m=+927.960166780"
Oct 31 01:20:07.552458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:07.552423    5419 generic.go:356] "Generic (PLEG): container finished" podID="d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" containerID="6a18f86b99ec85b60cbea4b6df3081f37fd87e0996a098e20ad809b9b2674062" exitCode=0
Oct 31 01:20:07.552874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:07.552514    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6" event={"ID":"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f","Type":"ContainerDied","Data":"6a18f86b99ec85b60cbea4b6df3081f37fd87e0996a098e20ad809b9b2674062"}
Oct 31 01:20:08.704084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.704060    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:08.765762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.765712    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c\" (UniqueName: \"kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c\") pod \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") "
Oct 31 01:20:08.765762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.765776    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca\") pod \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") "
Oct 31 01:20:08.766026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.765893    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca\") pod \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\" (UID: \"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f\") "
Oct 31 01:20:08.766276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.766244    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca" pod "d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" (UID: "d9daaaaf-047e-40b9-ab47-b01ad52e7e1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:08.766437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.766417    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:20:08.768066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.768042    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca" pod "d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" (UID: "d9daaaaf-047e-40b9-ab47-b01ad52e7e1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:08.768175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.768114    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c" pod "d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" (UID: "d9daaaaf-047e-40b9-ab47-b01ad52e7e1f"). InnerVolumeSpecName "kube-api-access-vjm2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:08.867810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.867780    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjm2c\" (UniqueName: \"kubernetes.io/projected/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-kube-api-access-vjm2c\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:20:08.867810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:08.867807    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:20:09.569192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:09.569119    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6" event={"ID":"d9daaaaf-047e-40b9-ab47-b01ad52e7e1f","Type":"ContainerDied","Data":"892f3cd51c7da3502c71d85b6381d41739e808a6497d66fb248be91c21ad03d6"}
Oct 31 01:20:09.569401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:09.569217    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="892f3cd51c7da3502c71d85b6381d41739e808a6497d66fb248be91c21ad03d6"
Oct 31 01:20:09.569401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:09.569294    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"
Oct 31 01:20:10.544089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.544054    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"]
Oct 31 01:20:10.544760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.544738    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" containerName="revoke-keys"
Oct 31 01:20:10.544805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.544764    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" containerName="revoke-keys"
Oct 31 01:20:10.544917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.544905    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" containerName="revoke-keys"
Oct 31 01:20:10.548808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.548788    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.601331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.601304    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:20:10.601542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.601352    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2d1088576145\""
Oct 31 01:20:10.650413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.650383    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:20:10.650601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.650384    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:20:10.650601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.650383    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:20:10.650601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.650542    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1984e9af4453\""
Oct 31 01:20:10.650601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.650573    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7qstt\""
Oct 31 01:20:10.652765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.652746    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"]
Oct 31 01:20:10.689746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.689711    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.689889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.689767    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltnb\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.689959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.689937    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.690003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.689977    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.791344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.791306    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.791344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.791349    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.791564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.791415    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.791564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.791441    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltnb\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.792087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.792063    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.792359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.792065    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.794122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.794072    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.847849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.847827    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltnb\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb\") pod \"payload-processing-external-model-67749c6fd4-wn8q2\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") " pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:10.862709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:10.862679    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:11.237940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:11.237778    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"]
Oct 31 01:20:11.240187 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:11.240115    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aed7cc5_9ca6_442d_869a_268b6c200fb0.slice/crio-becab1f53d5322e866e54fcb436214cdf463f027720c360380599cebfab5003d WatchSource:0}: Error finding container becab1f53d5322e866e54fcb436214cdf463f027720c360380599cebfab5003d: Status 404 returned error can't find the container with id becab1f53d5322e866e54fcb436214cdf463f027720c360380599cebfab5003d
Oct 31 01:20:11.591174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:11.591030    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2" event={"ID":"5aed7cc5-9ca6-442d-869a-268b6c200fb0","Type":"ContainerStarted","Data":"91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0"}
Oct 31 01:20:11.591174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:11.591102    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2" event={"ID":"5aed7cc5-9ca6-442d-869a-268b6c200fb0","Type":"ContainerStarted","Data":"becab1f53d5322e866e54fcb436214cdf463f027720c360380599cebfab5003d"}
Oct 31 01:20:11.591174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:11.591167    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:11.718171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:11.718101    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2" podStartSLOduration=1.7180873 podStartE2EDuration="1.7180873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:11.715850297 +0000 UTC m=+933.016806877" watchObservedRunningTime="2026-10-31 01:20:11.7180873 +0000 UTC m=+933.019043879"
Oct 31 01:20:14.582483 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:20:14.582452    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:20:16.046077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.046044    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"]
Oct 31 01:20:16.049978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.049959    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.102934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.102897    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:20:16.105444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.105426    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:20:16.105733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.105716    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:20:16.148519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.148478    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59dk\" (UniqueName: \"kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.148727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.148671    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.148727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.148714    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.152501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.152480    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:20:16.157193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.157171    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"]
Oct 31 01:20:16.249528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.249497    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.249727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.249535    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.249727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.249600    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59dk\" (UniqueName: \"kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.250296 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.250271    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.251993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.251973    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.306423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.306346    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59dk\" (UniqueName: \"kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk\") pod \"maas-api-revoke-sub-1b9e7cff-fgtt7\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.363268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.363229    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:16.842236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:16.842197    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"]
Oct 31 01:20:16.844169 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:16.844126    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29da3141_21b8_4155_a281_f54be4e4d868.slice/crio-b504c95cf76c02b85118a8419ce74ac40422a1150c5348cfb9e55a346a267185 WatchSource:0}: Error finding container b504c95cf76c02b85118a8419ce74ac40422a1150c5348cfb9e55a346a267185: Status 404 returned error can't find the container with id b504c95cf76c02b85118a8419ce74ac40422a1150c5348cfb9e55a346a267185
Oct 31 01:20:17.648594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:17.648556    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7" event={"ID":"29da3141-21b8-4155-a281-f54be4e4d868","Type":"ContainerStarted","Data":"1cd6ce17a9e3d09fedcf1e24226fa1a7fefdb5d9b94466664bbfbd4449e4fc65"}
Oct 31 01:20:17.649084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:17.648623    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7" event={"ID":"29da3141-21b8-4155-a281-f54be4e4d868","Type":"ContainerStarted","Data":"b504c95cf76c02b85118a8419ce74ac40422a1150c5348cfb9e55a346a267185"}
Oct 31 01:20:17.762489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:17.762438    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7" podStartSLOduration=1.762422551 podStartE2EDuration="1.762422551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:17.761976598 +0000 UTC m=+939.062933175" watchObservedRunningTime="2026-10-31 01:20:17.762422551 +0000 UTC m=+939.063379129"
Oct 31 01:20:18.656777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:18.656736    5419 generic.go:356] "Generic (PLEG): container finished" podID="29da3141-21b8-4155-a281-f54be4e4d868" containerID="1cd6ce17a9e3d09fedcf1e24226fa1a7fefdb5d9b94466664bbfbd4449e4fc65" exitCode=0
Oct 31 01:20:18.657193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:18.656818    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7" event={"ID":"29da3141-21b8-4155-a281-f54be4e4d868","Type":"ContainerDied","Data":"1cd6ce17a9e3d09fedcf1e24226fa1a7fefdb5d9b94466664bbfbd4449e4fc65"}
Oct 31 01:20:19.792687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.792660    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:19.901361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.901328    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca\") pod \"29da3141-21b8-4155-a281-f54be4e4d868\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") "
Oct 31 01:20:19.901543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.901375    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk\" (UniqueName: \"kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk\") pod \"29da3141-21b8-4155-a281-f54be4e4d868\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") "
Oct 31 01:20:19.901543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.901487    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca\") pod \"29da3141-21b8-4155-a281-f54be4e4d868\" (UID: \"29da3141-21b8-4155-a281-f54be4e4d868\") "
Oct 31 01:20:19.901777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.901746    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca" pod "29da3141-21b8-4155-a281-f54be4e4d868" (UID: "29da3141-21b8-4155-a281-f54be4e4d868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:19.902036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.902011    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:20:19.903718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.903697    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk" pod "29da3141-21b8-4155-a281-f54be4e4d868" (UID: "29da3141-21b8-4155-a281-f54be4e4d868"). InnerVolumeSpecName "kube-api-access-k59dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:19.903718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:19.903699    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca" pod "29da3141-21b8-4155-a281-f54be4e4d868" (UID: "29da3141-21b8-4155-a281-f54be4e4d868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:20.003497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:20.003466    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k59dk\" (UniqueName: \"kubernetes.io/projected/29da3141-21b8-4155-a281-f54be4e4d868-kube-api-access-k59dk\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:20:20.003497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:20.003494    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29da3141-21b8-4155-a281-f54be4e4d868-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:20:20.669289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:20.669261    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"
Oct 31 01:20:20.669474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:20.669261    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7" event={"ID":"29da3141-21b8-4155-a281-f54be4e4d868","Type":"ContainerDied","Data":"b504c95cf76c02b85118a8419ce74ac40422a1150c5348cfb9e55a346a267185"}
Oct 31 01:20:20.669474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:20.669385    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b504c95cf76c02b85118a8419ce74ac40422a1150c5348cfb9e55a346a267185"
Oct 31 01:20:22.602356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:22.602328    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:20:25.948151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:25.948106    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:20:28.898211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.898169    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"]
Oct 31 01:20:28.898659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.898642    5419 cpu_manager.go:414] "containerMap: removing container" podUID="29da3141-21b8-4155-a281-f54be4e4d868" containerName="revoke-keys"
Oct 31 01:20:28.898708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.898661    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29da3141-21b8-4155-a281-f54be4e4d868" containerName="revoke-keys"
Oct 31 01:20:28.898766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.898757    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="29da3141-21b8-4155-a281-f54be4e4d868" containerName="revoke-keys"
Oct 31 01:20:28.902242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.902226    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.997912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.997878    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdrz\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.997925    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998022    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998054    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998078    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998166    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998237    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998253    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:28.998321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:28.998317    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.006618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.006591    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"]
Oct 31 01:20:29.098864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.098832    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.098864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.098864    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.098885    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.098937    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdrz\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.098967    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.098988    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099008    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099037    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099335    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099709    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099853    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099890    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.099959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.099913    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.101635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.101613    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.101737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.101646    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.155873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.155800    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.156037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.155946    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdrz\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-jg8xt\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.215051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.215018    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:29.644558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.644532    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"]
Oct 31 01:20:29.645909 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:29.645882    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142e8bd0_8218_415c_accd_f6a82f66617d.slice/crio-58aea93ef0429103468d87f5a35d5829b8ec05241699c81c896b0495b515f326 WatchSource:0}: Error finding container 58aea93ef0429103468d87f5a35d5829b8ec05241699c81c896b0495b515f326: Status 404 returned error can't find the container with id 58aea93ef0429103468d87f5a35d5829b8ec05241699c81c896b0495b515f326
Oct 31 01:20:29.648314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.648284    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:29.648395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.648350    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:29.648395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.648378    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:29.734064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.734011    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" event={"ID":"142e8bd0-8218-415c-accd-f6a82f66617d","Type":"ContainerStarted","Data":"31b23d224f3083d2315881d354719713bbca50d22c5926897ff1010e93f05d35"}
Oct 31 01:20:29.734249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.734093    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" event={"ID":"142e8bd0-8218-415c-accd-f6a82f66617d","Type":"ContainerStarted","Data":"58aea93ef0429103468d87f5a35d5829b8ec05241699c81c896b0495b515f326"}
Oct 31 01:20:29.850105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:29.850036    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podStartSLOduration=1.850019256 podStartE2EDuration="1.850019256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:29.849564408 +0000 UTC m=+951.150520986" watchObservedRunningTime="2026-10-31 01:20:29.850019256 +0000 UTC m=+951.150975834"
Oct 31 01:20:30.215782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.215745    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:30.217389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.217361    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:30.217527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.217414    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:30.444325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.444291    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"]
Oct 31 01:20:30.448172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.448130    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.500505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.500464    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-00e840-openshift-default-dockercfg-2nfqp\""
Oct 31 01:20:30.553501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.553470    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"]
Oct 31 01:20:30.616534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616500    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616550    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616575    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616687    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gc9t\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616722    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616750    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616818    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616867    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.616941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.616925    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.717800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.717764    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.717991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.717812    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.717991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.717873    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.717991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.717930    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.717991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.717980    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718115    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718197    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gc9t\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718250    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718296    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718466 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718399    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718455    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718641    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718664    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.718770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.718734    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.720555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.720527    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.720841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.720823    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.805208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.795903    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gc9t\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:30.805208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:30.799700    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:31.061725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.061639    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:31.215717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.215670    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:31.215917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.215738    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:31.511792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.511767    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"]
Oct 31 01:20:31.513407 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:31.513380    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefd1519_70d7_4a57_8f72_f6a32b7e670b.slice/crio-0124212f546c228ff2d9f30862d9eb59d78f7e6fc857ea82f134f9f63e658836 WatchSource:0}: Error finding container 0124212f546c228ff2d9f30862d9eb59d78f7e6fc857ea82f134f9f63e658836: Status 404 returned error can't find the container with id 0124212f546c228ff2d9f30862d9eb59d78f7e6fc857ea82f134f9f63e658836
Oct 31 01:20:31.515696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.515655    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:31.515794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.515744    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:31.515794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.515772    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:31.747774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.747734    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" event={"ID":"defd1519-70d7-4a57-8f72-f6a32b7e670b","Type":"ContainerStarted","Data":"78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580"}
Oct 31 01:20:31.747965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.747798    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" event={"ID":"defd1519-70d7-4a57-8f72-f6a32b7e670b","Type":"ContainerStarted","Data":"0124212f546c228ff2d9f30862d9eb59d78f7e6fc857ea82f134f9f63e658836"}
Oct 31 01:20:31.871077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:31.870978    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podStartSLOduration=1.870964303 podStartE2EDuration="1.870964303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:31.870301674 +0000 UTC m=+953.171258287" watchObservedRunningTime="2026-10-31 01:20:31.870964303 +0000 UTC m=+953.171920898"
Oct 31 01:20:32.062100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:32.062063    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:32.063567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:32.063545    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:32.063622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:32.063590    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:32.217765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:32.217669    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:32.218429 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:32.218393    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:33.062865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:33.062826    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:33.063047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:33.062892    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:33.216651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:33.216616    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:33.216999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:33.216968    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:34.062105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:34.062062    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:34.062538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:34.062122    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:34.220591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:34.220558    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:34.220863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:34.220837    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:35.062484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:35.062452    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:35.063263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:35.062951    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:35.218164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:35.217600    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:35.218164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:35.217663    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:36.063074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:36.063036    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:36.063518 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:36.063094    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:36.215703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:36.215651    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:36.215913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:36.215714    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:37.062641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:37.062605    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:37.062817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:37.062668    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:37.215377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:37.215344    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 01:20:37.215805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:37.215399    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 01:20:38.062568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:38.062532    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:20:38.062762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:38.062614    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 01:20:38.425629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:38.425526    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:38.426562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:38.426533    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:38.427527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:38.427502    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:20:39.066850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:39.066821    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:39.067365 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:39.067352    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:39.068148 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:39.068111    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 01:20:40.713165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:40.712334    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:20:40.713165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:40.712436    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:20:44.051821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.051786    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"]
Oct 31 01:20:44.056815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.056799    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.187759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.187716    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.187922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.187805    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.187922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.187839    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.187995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.187935    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmrt\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.188055 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.188036    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.188107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.188092    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.188182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.188123    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.188244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.188228    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.188357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.188342    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.258383 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.258345    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"]
Oct 31 01:20:44.289789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.289748    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmrt\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.289982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.289815    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.289982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.289854    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.289982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.289877    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.289982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.289906    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.289984    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290044    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290373    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290405    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290426    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290435    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290531    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290693    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.290908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.290885    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.292585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.292561    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.292758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.292739    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.346901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.346826    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.347040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.346898    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmrt\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5tdsb\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.369937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.369908    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:44.591762 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:20:44.591612    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:20:44.797479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.797451    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"]
Oct 31 01:20:44.798818 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:44.798793    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4ca327_19c4_4ab7_87f2_9fbf8179cd61.slice/crio-5d6a02d7542ac3fdf3f1742ca27ab0cfd26648aa31d4dd804121a431c7b482ee WatchSource:0}: Error finding container 5d6a02d7542ac3fdf3f1742ca27ab0cfd26648aa31d4dd804121a431c7b482ee: Status 404 returned error can't find the container with id 5d6a02d7542ac3fdf3f1742ca27ab0cfd26648aa31d4dd804121a431c7b482ee
Oct 31 01:20:44.801106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.801076    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:44.801207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.801158    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:44.801207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.801188    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:44.844756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:44.844731    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" event={"ID":"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61","Type":"ContainerStarted","Data":"5d6a02d7542ac3fdf3f1742ca27ab0cfd26648aa31d4dd804121a431c7b482ee"}
Oct 31 01:20:45.852664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:45.852622    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" event={"ID":"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61","Type":"ContainerStarted","Data":"7cb95175c97d691a272dfc8cff4f29969d4dd2da91f1a7aefbc9ee3b253a5643"}
Oct 31 01:20:46.070261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:46.070210    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podStartSLOduration=2.070197029 podStartE2EDuration="2.070197029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:46.018828054 +0000 UTC m=+967.319784631" watchObservedRunningTime="2026-10-31 01:20:46.070197029 +0000 UTC m=+967.371153606"
Oct 31 01:20:46.370278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:46.370244    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:46.371412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:46.371391    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:46.371468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:46.371438    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:47.370641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:47.370608    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:47.371070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:47.370667    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:48.371133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:48.371095    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:48.371566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:48.371178    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:49.370794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:49.370759    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:49.370980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:49.370815    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:50.370969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:50.370931    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:50.371428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:50.371002    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:51.370856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:51.370816    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:51.371051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:51.370885    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:52.370993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:52.370949    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:20:52.371256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:52.371001    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 01:20:53.375321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:53.375291    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:53.375804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:53.375773    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:53.376391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:53.376371    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:20:55.839904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:55.839869    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:20:59.159270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.159232    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"]
Oct 31 01:20:59.163030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.163013    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260285    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260409    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260448    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260549    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260627    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79gnb\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260655    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260683    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260716    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.260909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.260748    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.267871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.267843    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"]
Oct 31 01:20:59.361503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361463    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79gnb\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361506    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361532    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361561    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361588    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361660    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361775    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361811    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.361986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.361849    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.362176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.362031    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.362249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.362203    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.362249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.362224    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.362425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.362291    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.362597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.362574    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.364084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.364062    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.364336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.364317    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.419090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.419012    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.419307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.419280    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79gnb\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb\") pod \"maas-default-gateway-openshift-default-75fd9f5d47-5ks26\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.477119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.477084    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:20:59.903150 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.903111    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"]
Oct 31 01:20:59.905116 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:20:59.905090    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f52ca6_06ad_4fcd_b758_b530cf92c846.slice/crio-1b3f04b4cc9afc5371cf90a0eaa00eb68da4519925e6e2dd693ebd542b594c06 WatchSource:0}: Error finding container 1b3f04b4cc9afc5371cf90a0eaa00eb68da4519925e6e2dd693ebd542b594c06: Status 404 returned error can't find the container with id 1b3f04b4cc9afc5371cf90a0eaa00eb68da4519925e6e2dd693ebd542b594c06
Oct 31 01:20:59.907430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.907404    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:59.907497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.907470    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:59.907534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.907505    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:20:59.980202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:20:59.975484    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" event={"ID":"d0f52ca6-06ad-4fcd-b758-b530cf92c846","Type":"ContainerStarted","Data":"1b3f04b4cc9afc5371cf90a0eaa00eb68da4519925e6e2dd693ebd542b594c06"}
Oct 31 01:21:01.007607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:01.007573    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" event={"ID":"d0f52ca6-06ad-4fcd-b758-b530cf92c846","Type":"ContainerStarted","Data":"09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb"}
Oct 31 01:21:01.197620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:01.197548    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podStartSLOduration=2.197524378 podStartE2EDuration="2.197524378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:01.177341469 +0000 UTC m=+982.478298046" watchObservedRunningTime="2026-10-31 01:21:01.197524378 +0000 UTC m=+982.498480955"
Oct 31 01:21:01.492164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:01.485521    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:21:01.499317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:01.499278    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:01.499629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:01.499597    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:02.489582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:02.489544    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:02.490212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:02.490179    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:03.493174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:03.485594    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:03.493174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:03.485654    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:04.494165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:04.487688    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:04.494165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:04.487769    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:05.495165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:05.485517    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:05.495165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:05.485580    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:06.477414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:06.477382    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:06.477598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:06.477435    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:07.478438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:07.478402    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 01:21:07.478831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:07.478467    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 01:21:08.482203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:08.482173    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:21:08.482683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:08.482566    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:21:08.483300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:08.483283    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:21:14.581726 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:14.581687    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:21:27.988629 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:27.988532    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1984e9af4453: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:27.988629 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:27.988570    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:27.989308 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:27.988651    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:21:28.488631009 +0000 UTC m=+1009.789587609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:28.496629 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:28.496599    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1984e9af4453: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:28.496629 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:28.496630    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:28.496831 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:28.496695    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:21:29.496680887 +0000 UTC m=+1010.797637443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:29.507664 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:29.507631    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1984e9af4453: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:29.507664 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:29.507659    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:29.508199 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:29.507717    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:21:31.507703428 +0000 UTC m=+1012.808659984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:31.528210 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:31.528171    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-1984e9af4453: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:31.528210 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:31.528205    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:31.528745 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:31.528277    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:21:35.52825803 +0000 UTC m=+1016.829214600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-1984e9af4453" not found
Oct 31 01:21:39.016900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.016866    5419 scope.go:122] "RemoveContainer" containerID="8156c152a86d049b3ef604912bfa351ad8f1464e15595bcc3c5ae5cfe3ed3f1d"
Oct 31 01:21:39.029053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.028999    5419 scope.go:122] "RemoveContainer" containerID="5a647bddbd5ca6797f89c3c545a23f9896de2645020484696806cf0196169742"
Oct 31 01:21:39.390745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.390653    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"]
Oct 31 01:21:39.396086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.396062    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.422428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.422393    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.422571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.422454    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kjx\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.422571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.422548    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.422644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.422588    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.499491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.499463    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"]
Oct 31 01:21:39.524205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.524174    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.524376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.524246    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kjx\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.524376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.524295    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.524376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.524324    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.528158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.525572    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.528158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.525897    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.533202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.533178    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.581811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.581784    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kjx\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx\") pod \"payload-processing-external-model-5dcfb47976-sl56j\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:39.709277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:39.709189    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:40.090304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:40.090277    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"]
Oct 31 01:21:40.091986 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:21:40.091960    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2cbf74_7196_4cba_944c_0ba5b3e0f074.slice/crio-d92bb09986aee7a67cc25d701be93a35128e271d2f44594bc65b734fb38f91ef WatchSource:0}: Error finding container d92bb09986aee7a67cc25d701be93a35128e271d2f44594bc65b734fb38f91ef: Status 404 returned error can't find the container with id d92bb09986aee7a67cc25d701be93a35128e271d2f44594bc65b734fb38f91ef
Oct 31 01:21:40.266849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:40.266816    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" event={"ID":"8a2cbf74-7196-4cba-944c-0ba5b3e0f074","Type":"ContainerStarted","Data":"165877b9d73af0445663b65ccc78b2c4a0219884db224823e5641ba22c430f5a"}
Oct 31 01:21:40.267021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:40.266874    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" event={"ID":"8a2cbf74-7196-4cba-944c-0ba5b3e0f074","Type":"ContainerStarted","Data":"d92bb09986aee7a67cc25d701be93a35128e271d2f44594bc65b734fb38f91ef"}
Oct 31 01:21:40.267021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:40.266907    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:21:40.382853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:40.382752    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podStartSLOduration=1.38273713 podStartE2EDuration="1.38273713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:40.380991816 +0000 UTC m=+1021.681948421" watchObservedRunningTime="2026-10-31 01:21:40.38273713 +0000 UTC m=+1021.683693706"
Oct 31 01:21:44.581014 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:21:44.580976    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:21:45.645810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:45.645774    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:21:51.274837 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:21:51.274799    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:01.273825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:01.273790    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:09.191514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.191479    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"]
Oct 31 01:22:09.195319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.195303    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.335948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.335913    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.335955    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336053    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336120    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336226    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336254    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336273    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knc2g\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336290    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.336411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.336398    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.401939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.401899    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"]
Oct 31 01:22:09.437829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.437794    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.437837    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.437868    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438059    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438117    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438170    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438196    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knc2g\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438227    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438281    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438404    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438648    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438772    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.438973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.438905    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.439214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.439189    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.440921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.440893    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.441031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.440996    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.495902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.495870    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.496051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.495935    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knc2g\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.508817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.508791    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:09.955713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.955666    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"]
Oct 31 01:22:09.962112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.961040    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:22:09.962112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.961123    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:22:09.962112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:09.961187    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:22:10.104155 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.104102    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"]
Oct 31 01:22:10.108808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.108781    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.162358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.162274    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8f6b5b-openshift-default-dockercfg-6t9gt\""
Oct 31 01:22:10.214329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.214251    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"]
Oct 31 01:22:10.251416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251386    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251419    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251458    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hjd\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251542    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251563    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251611    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251696    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251723    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.251810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.251747    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352355    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352401    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352422    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352459    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352484    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352521    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hjd\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352577    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352605    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352668    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.352955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.352903    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.353122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.353094    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.353302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.353223    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.353302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.353330    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.353738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.353712    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.355041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.355018    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.355273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.355249    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.412126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.412097    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.412351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.412237    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hjd\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.424262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.424235    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:10.532029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.531992    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" event={"ID":"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f","Type":"ContainerStarted","Data":"4abdd112adfe55d4ef7b9f82b7462f208a3256e7aff7d3f1a89d4db41124b993"}
Oct 31 01:22:10.532198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.532061    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" event={"ID":"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f","Type":"ContainerStarted","Data":"61152922c5b29d47ca27dd50124ab59d739d03aa87f4493d515758f8b56c08e3"}
Oct 31 01:22:10.651252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.651199    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podStartSLOduration=1.651184572 podStartE2EDuration="1.651184572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:10.650227562 +0000 UTC m=+1051.951184140" watchObservedRunningTime="2026-10-31 01:22:10.651184572 +0000 UTC m=+1051.952141148"
Oct 31 01:22:10.905200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.903965    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"]
Oct 31 01:22:10.906197 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:22:10.906160    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5daad64_325f_486e_8a64_166579b9dac8.slice/crio-fcea05a450a1c4d761096f1923cbbad69703947f9cb2f78a2608d646639ce5ad WatchSource:0}: Error finding container fcea05a450a1c4d761096f1923cbbad69703947f9cb2f78a2608d646639ce5ad: Status 404 returned error can't find the container with id fcea05a450a1c4d761096f1923cbbad69703947f9cb2f78a2608d646639ce5ad
Oct 31 01:22:10.908760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.908730    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:22:10.908847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.908797    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:22:10.908847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:10.908825    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:22:11.273871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.273828    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:11.509488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.509451    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:11.510933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.510905    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:22:11.511063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.510959    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:22:11.538836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.538747    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" event={"ID":"f5daad64-325f-486e-8a64-166579b9dac8","Type":"ContainerStarted","Data":"7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251"}
Oct 31 01:22:11.538836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.538808    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" event={"ID":"f5daad64-325f-486e-8a64-166579b9dac8","Type":"ContainerStarted","Data":"fcea05a450a1c4d761096f1923cbbad69703947f9cb2f78a2608d646639ce5ad"}
Oct 31 01:22:11.662213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:11.662163    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podStartSLOduration=1.662129344 podStartE2EDuration="1.662129344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:11.660344377 +0000 UTC m=+1052.961300982" watchObservedRunningTime="2026-10-31 01:22:11.662129344 +0000 UTC m=+1052.963085921"
Oct 31 01:22:12.425398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:12.425360    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:12.427097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:12.427075    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:12.427288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:12.427267    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:22:12.515744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:12.515591    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:22:12.515744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:12.515678    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:22:13.425556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:13.425515    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:13.426100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:13.425577    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:22:13.515520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:13.515484    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:22:13.515812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:13.515778    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:22:14.424716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:14.424678    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:14.424953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:14.424747    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:22:14.513165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:14.510523    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:22:14.513165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:14.510583    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:22:14.630193 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:14.630155    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:22:14.756546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:14.756511    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:22:15.425585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:15.425548    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:15.425771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:15.425615    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:22:15.509594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:15.509553    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:22:15.509923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:15.509882    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:22:16.424664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:16.424631    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:16.425069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:16.424686    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:22:16.509867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:16.509830    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 01:22:16.510049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:16.509901    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 01:22:17.425428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:17.425388    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 01:22:17.425835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:17.425458    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 01:22:17.514298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:17.514261    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:17.514656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:17.514633    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:17.515254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:17.515233    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:22:18.429749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:18.429717    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:18.430307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:18.430037    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:18.431003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:18.430965    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:22:21.273166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:21.273113    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:31.273888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:31.273850    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:33.993800 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:33.993763    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d1088576145: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:33.993800 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:33.993792    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:33.996462 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:33.993873    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:22:34.493852168 +0000 UTC m=+1075.794808724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-2d1088576145" not found
Oct 31 01:22:34.498733 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:34.498692    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d1088576145: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:34.498733 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:34.498722    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:34.498992 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:34.498795    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:22:35.498776206 +0000 UTC m=+1076.799732776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-2d1088576145" not found
Oct 31 01:22:35.509008 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:35.508967    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d1088576145: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:35.509008 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:35.508996    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:35.509559 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:35.509062    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:22:37.509042678 +0000 UTC m=+1078.809999247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-2d1088576145" not found
Oct 31 01:22:37.529764 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:37.529730    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d1088576145: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:37.529764 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:37.529755    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:37.530223 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:37.529817    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:22:41.529802828 +0000 UTC m=+1082.830759383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-2d1088576145" not found
Oct 31 01:22:39.055843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:39.055810    5419 scope.go:122] "RemoveContainer" containerID="9c57725e493ad50062a93c7ff4500a711301a6c621a3f53b944169944dbee274"
Oct 31 01:22:39.066905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:39.066885    5419 scope.go:122] "RemoveContainer" containerID="d884f8dfedef90c84b323e90412d181d50b51f1dcf4501cfb9e92a364c5ba9b7"
Oct 31 01:22:39.075759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:39.075736    5419 scope.go:122] "RemoveContainer" containerID="a329d9917bcaa5012f353150f4f08353e9b966f59a51a880f9770b576804627d"
Oct 31 01:22:41.273071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:41.273037    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:41.583329 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:41.583250    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d1088576145: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:41.583329 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:41.583277    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:41.583517 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:41.583344    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:22:49.583329054 +0000 UTC m=+1090.884285609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-2d1088576145" not found
Oct 31 01:22:44.297458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.297421    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"]
Oct 31 01:22:44.302997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.302972    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.405261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.405224    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:22:44.405261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.405234    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:22:44.405467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.405303    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:22:44.405467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.405338    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:22:44.407892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.407875    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"]
Oct 31 01:22:44.431210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.431172    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.431345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.431215    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.431345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.431317    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngl7g\" (UniqueName: \"kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.532011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.531969    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngl7g\" (UniqueName: \"kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.532249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.532212    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.532249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.532242    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.532864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.532841    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.534812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.534788    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.582113 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:44.582027    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:22:44.588726 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.588707    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngl7g\" (UniqueName: \"kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g\") pod \"maas-api-revoke-sub-5e7ff7ac-kk2w2\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.615740 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.615708    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:44.989200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:44.989172    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"]
Oct 31 01:22:44.990844 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:22:44.990818    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9391ce64_0dd5_46d0_a7ec_69ca16ece995.slice/crio-da547d8a970bc8f35bf96e4f57d5ecd6ae23134a95c49e807cf8abce9dca3039 WatchSource:0}: Error finding container da547d8a970bc8f35bf96e4f57d5ecd6ae23134a95c49e807cf8abce9dca3039: Status 404 returned error can't find the container with id da547d8a970bc8f35bf96e4f57d5ecd6ae23134a95c49e807cf8abce9dca3039
Oct 31 01:22:45.771587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:45.771551    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2" event={"ID":"9391ce64-0dd5-46d0-a7ec-69ca16ece995","Type":"ContainerStarted","Data":"74db30d4c5cdd8d4a7482a438786f3807b283a44609ebdf76dd534d76a57f52f"}
Oct 31 01:22:45.772078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:45.771643    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2" event={"ID":"9391ce64-0dd5-46d0-a7ec-69ca16ece995","Type":"ContainerStarted","Data":"da547d8a970bc8f35bf96e4f57d5ecd6ae23134a95c49e807cf8abce9dca3039"}
Oct 31 01:22:45.877309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:45.877279    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"]
Oct 31 01:22:45.878721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:45.878684    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:22:45.888868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:45.888822    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2" podStartSLOduration=1.888809569 podStartE2EDuration="1.888809569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:45.885844655 +0000 UTC m=+1087.186801232" watchObservedRunningTime="2026-10-31 01:22:45.888809569 +0000 UTC m=+1087.189766146"
Oct 31 01:22:46.779127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:46.779088    5419 generic.go:356] "Generic (PLEG): container finished" podID="9391ce64-0dd5-46d0-a7ec-69ca16ece995" containerID="74db30d4c5cdd8d4a7482a438786f3807b283a44609ebdf76dd534d76a57f52f" exitCode=0
Oct 31 01:22:46.779696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:46.779177    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2" event={"ID":"9391ce64-0dd5-46d0-a7ec-69ca16ece995","Type":"ContainerDied","Data":"74db30d4c5cdd8d4a7482a438786f3807b283a44609ebdf76dd534d76a57f52f"}
Oct 31 01:22:47.932786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:47.932759    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:48.077552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.077466    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g\" (UniqueName: \"kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g\") pod \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") "
Oct 31 01:22:48.077718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.077563    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca\") pod \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") "
Oct 31 01:22:48.077718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.077658    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca\") pod \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\" (UID: \"9391ce64-0dd5-46d0-a7ec-69ca16ece995\") "
Oct 31 01:22:48.077986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.077953    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca" pod "9391ce64-0dd5-46d0-a7ec-69ca16ece995" (UID: "9391ce64-0dd5-46d0-a7ec-69ca16ece995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:48.078583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.078557    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:48.079907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.079884    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g" pod "9391ce64-0dd5-46d0-a7ec-69ca16ece995" (UID: "9391ce64-0dd5-46d0-a7ec-69ca16ece995"). InnerVolumeSpecName "kube-api-access-ngl7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:48.080070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.080052    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca" pod "9391ce64-0dd5-46d0-a7ec-69ca16ece995" (UID: "9391ce64-0dd5-46d0-a7ec-69ca16ece995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:22:48.180152 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.180115    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9391ce64-0dd5-46d0-a7ec-69ca16ece995-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:48.180310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.180170    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngl7g\" (UniqueName: \"kubernetes.io/projected/9391ce64-0dd5-46d0-a7ec-69ca16ece995-kube-api-access-ngl7g\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:48.791734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.791701    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2" event={"ID":"9391ce64-0dd5-46d0-a7ec-69ca16ece995","Type":"ContainerDied","Data":"da547d8a970bc8f35bf96e4f57d5ecd6ae23134a95c49e807cf8abce9dca3039"}
Oct 31 01:22:48.791734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.791721    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"
Oct 31 01:22:48.791947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:48.791748    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da547d8a970bc8f35bf96e4f57d5ecd6ae23134a95c49e807cf8abce9dca3039"
Oct 31 01:22:49.344445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:49.344414    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:22:49.597251 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:49.597175    5419 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2d1088576145: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:49.597251 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:49.597199    5419 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2: secret "agc-cred-2d1088576145" not found
Oct 31 01:22:49.597251 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:22:49.597250    5419 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials podName:5aed7cc5-9ca6-442d-869a-268b6c200fb0 nodeName:}" failed. No retries permitted until 2026-10-31 01:23:05.597236649 +0000 UTC m=+1106.898193217 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials") pod "payload-processing-external-model-67749c6fd4-wn8q2" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0") : secret "agc-cred-2d1088576145" not found
Oct 31 01:22:50.879410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:50.879368    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing" containerID="cri-o://165877b9d73af0445663b65ccc78b2c4a0219884db224823e5641ba22c430f5a" gracePeriod=25
Oct 31 01:22:51.952235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:51.950557    5419 generic.go:356] "Generic (PLEG): container finished" podID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerID="165877b9d73af0445663b65ccc78b2c4a0219884db224823e5641ba22c430f5a" exitCode=0
Oct 31 01:22:51.952235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:51.950669    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" event={"ID":"8a2cbf74-7196-4cba-944c-0ba5b3e0f074","Type":"ContainerDied","Data":"165877b9d73af0445663b65ccc78b2c4a0219884db224823e5641ba22c430f5a"}
Oct 31 01:22:52.083689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.081235    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:22:52.083689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.081606    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 01:22:52.279165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.267266    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:22:52.315511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.315477    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:22:52.371024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.370976    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume\") pod \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") "
Oct 31 01:22:52.371398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.371376    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials\") pod \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") "
Oct 31 01:22:52.371630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.371609    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx\") pod \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") "
Oct 31 01:22:52.371816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.371790    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume\") pod \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\" (UID: \"8a2cbf74-7196-4cba-944c-0ba5b3e0f074\") "
Oct 31 01:22:52.373738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.373706    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume" pod "8a2cbf74-7196-4cba-944c-0ba5b3e0f074" (UID: "8a2cbf74-7196-4cba-944c-0ba5b3e0f074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:52.377675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.377644    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume" pod "8a2cbf74-7196-4cba-944c-0ba5b3e0f074" (UID: "8a2cbf74-7196-4cba-944c-0ba5b3e0f074"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:22:52.400173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.397686    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx" pod "8a2cbf74-7196-4cba-944c-0ba5b3e0f074" (UID: "8a2cbf74-7196-4cba-944c-0ba5b3e0f074"). InnerVolumeSpecName "kube-api-access-d9kjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:52.404297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.404257    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials" pod "8a2cbf74-7196-4cba-944c-0ba5b3e0f074" (UID: "8a2cbf74-7196-4cba-944c-0ba5b3e0f074"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:22:52.440448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.438220    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:22:52.486913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.485748    5419 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-provider-credentials\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:52.486913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.485784    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kjx\" (UniqueName: \"kubernetes.io/projected/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-kube-api-access-d9kjx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:52.486913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.485801    5419 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-routing-overlay-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:52.486913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.485820    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2cbf74-7196-4cba-944c-0ba5b3e0f074-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:22:52.794884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.786241    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:22:52.794884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.793249    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:22:52.962077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.962025    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j" event={"ID":"8a2cbf74-7196-4cba-944c-0ba5b3e0f074","Type":"ContainerDied","Data":"d92bb09986aee7a67cc25d701be93a35128e271d2f44594bc65b734fb38f91ef"}
Oct 31 01:22:52.962732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.962712    5419 scope.go:122] "RemoveContainer" containerID="165877b9d73af0445663b65ccc78b2c4a0219884db224823e5641ba22c430f5a"
Oct 31 01:22:52.963118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:52.963089    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"
Oct 31 01:22:53.171384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:53.171231    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"]
Oct 31 01:22:53.186163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:53.174354    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-sl56j"]
Oct 31 01:22:54.837156 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:54.837101    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" path="/var/lib/kubelet/pods/8a2cbf74-7196-4cba-944c-0ba5b3e0f074/volumes"
Oct 31 01:22:58.723224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:22:58.723188    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"]
Oct 31 01:23:00.655340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655309    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"]
Oct 31 01:23:00.655795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655781    5419 cpu_manager.go:414] "containerMap: removing container" podUID="9391ce64-0dd5-46d0-a7ec-69ca16ece995" containerName="revoke-keys"
Oct 31 01:23:00.655842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655796    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9391ce64-0dd5-46d0-a7ec-69ca16ece995" containerName="revoke-keys"
Oct 31 01:23:00.655842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655816    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing"
Oct 31 01:23:00.655842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655821    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing"
Oct 31 01:23:00.655932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655902    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="9391ce64-0dd5-46d0-a7ec-69ca16ece995" containerName="revoke-keys"
Oct 31 01:23:00.655932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.655916    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2cbf74-7196-4cba-944c-0ba5b3e0f074" containerName="payload-processing"
Oct 31 01:23:00.659367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.659351    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.763419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.763376    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:23:00.763608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.763463    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:23:00.763608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.763481    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:23:00.763608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.763480    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:23:00.765989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.765967    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"]
Oct 31 01:23:00.788022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.787993    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47jx\" (UniqueName: \"kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.788177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.788055    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.788230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.788214    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.889500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.889465    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.889679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.889580    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47jx\" (UniqueName: \"kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.889679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.889630    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.890275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.890240    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.891932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.891910    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.946438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.946404    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47jx\" (UniqueName: \"kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx\") pod \"maas-api-revoke-sub-384310bc-rhk2p\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:00.971915 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:00.971895    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:01.348408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:01.348384    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"]
Oct 31 01:23:01.349445 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:01.349415    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda093b4d2_d792_4541_a260_c40380c86fa1.slice/crio-ce3c9765c5ef0dc7f15db6c2243a183040248e885a271899f1fd70392d7eddca WatchSource:0}: Error finding container ce3c9765c5ef0dc7f15db6c2243a183040248e885a271899f1fd70392d7eddca: Status 404 returned error can't find the container with id ce3c9765c5ef0dc7f15db6c2243a183040248e885a271899f1fd70392d7eddca
Oct 31 01:23:02.026255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:02.026216    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p" event={"ID":"a093b4d2-d792-4541-a260-c40380c86fa1","Type":"ContainerStarted","Data":"ec2c4e3e983da02bedfbea8c9d2607f0c665c1cb5628a4a7c135f7574c031bba"}
Oct 31 01:23:02.026697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:02.026268    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p" event={"ID":"a093b4d2-d792-4541-a260-c40380c86fa1","Type":"ContainerStarted","Data":"ce3c9765c5ef0dc7f15db6c2243a183040248e885a271899f1fd70392d7eddca"}
Oct 31 01:23:02.139567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:02.139517    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p" podStartSLOduration=2.139502473 podStartE2EDuration="2.139502473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:02.138165762 +0000 UTC m=+1103.439122333" watchObservedRunningTime="2026-10-31 01:23:02.139502473 +0000 UTC m=+1103.440459049"
Oct 31 01:23:03.033175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:03.033117    5419 generic.go:356] "Generic (PLEG): container finished" podID="a093b4d2-d792-4541-a260-c40380c86fa1" containerID="ec2c4e3e983da02bedfbea8c9d2607f0c665c1cb5628a4a7c135f7574c031bba" exitCode=0
Oct 31 01:23:03.033545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:03.033190    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p" event={"ID":"a093b4d2-d792-4541-a260-c40380c86fa1","Type":"ContainerDied","Data":"ec2c4e3e983da02bedfbea8c9d2607f0c665c1cb5628a4a7c135f7574c031bba"}
Oct 31 01:23:03.724463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:03.724406    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2" podUID="5aed7cc5-9ca6-442d-869a-268b6c200fb0" containerName="payload-processing" containerID="cri-o://91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0" gracePeriod=25
Oct 31 01:23:04.031038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.031014    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:23:04.039479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.039453    5419 generic.go:356] "Generic (PLEG): container finished" podID="5aed7cc5-9ca6-442d-869a-268b6c200fb0" containerID="91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0" exitCode=0
Oct 31 01:23:04.039818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.039493    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"
Oct 31 01:23:04.039818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.039537    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2" event={"ID":"5aed7cc5-9ca6-442d-869a-268b6c200fb0","Type":"ContainerDied","Data":"91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0"}
Oct 31 01:23:04.039818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.039589    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2" event={"ID":"5aed7cc5-9ca6-442d-869a-268b6c200fb0","Type":"ContainerDied","Data":"becab1f53d5322e866e54fcb436214cdf463f027720c360380599cebfab5003d"}
Oct 31 01:23:04.039818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.039614    5419 scope.go:122] "RemoveContainer" containerID="91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0"
Oct 31 01:23:04.050804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.050782    5419 scope.go:122] "RemoveContainer" containerID="91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0"
Oct 31 01:23:04.051115 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:23:04.051088    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0\": container with ID starting with 91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0 not found: ID does not exist" containerID="91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0"
Oct 31 01:23:04.051259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.051116    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0"} err="failed to get container status \"91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0\": rpc error: code = NotFound desc = could not find container \"91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0\": container with ID starting with 91f8c3d51aa9a7f44c17c4a4c2407de994aa86415c0674b9145b2b7d112742c0 not found: ID does not exist"
Oct 31 01:23:04.134799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.134766    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials\") pod \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") "
Oct 31 01:23:04.134943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.134805    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume\") pod \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") "
Oct 31 01:23:04.134943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.134888    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume\") pod \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") "
Oct 31 01:23:04.135043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.135026    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb\") pod \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\" (UID: \"5aed7cc5-9ca6-442d-869a-268b6c200fb0\") "
Oct 31 01:23:04.135265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.135230    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume" pod "5aed7cc5-9ca6-442d-869a-268b6c200fb0" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:04.135377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.135298    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume" pod "5aed7cc5-9ca6-442d-869a-268b6c200fb0" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:04.135557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.135536    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.135669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.135559    5419 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5aed7cc5-9ca6-442d-869a-268b6c200fb0-routing-overlay-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.137295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.137274    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials" pod "5aed7cc5-9ca6-442d-869a-268b6c200fb0" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:04.137375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.137354    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb" pod "5aed7cc5-9ca6-442d-869a-268b6c200fb0" (UID: "5aed7cc5-9ca6-442d-869a-268b6c200fb0"). InnerVolumeSpecName "kube-api-access-xltnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:04.167061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.167036    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:04.236802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.236775    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xltnb\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-kube-api-access-xltnb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.236802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.236799    5419 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5aed7cc5-9ca6-442d-869a-268b6c200fb0-provider-credentials\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.338238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.338158    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx\" (UniqueName: \"kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx\") pod \"a093b4d2-d792-4541-a260-c40380c86fa1\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") "
Oct 31 01:23:04.338391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.338240    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca\") pod \"a093b4d2-d792-4541-a260-c40380c86fa1\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") "
Oct 31 01:23:04.338391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.338306    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca\") pod \"a093b4d2-d792-4541-a260-c40380c86fa1\" (UID: \"a093b4d2-d792-4541-a260-c40380c86fa1\") "
Oct 31 01:23:04.338696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.338665    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca" pod "a093b4d2-d792-4541-a260-c40380c86fa1" (UID: "a093b4d2-d792-4541-a260-c40380c86fa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:04.340661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.340635    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca" pod "a093b4d2-d792-4541-a260-c40380c86fa1" (UID: "a093b4d2-d792-4541-a260-c40380c86fa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:04.340806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.340781    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx" pod "a093b4d2-d792-4541-a260-c40380c86fa1" (UID: "a093b4d2-d792-4541-a260-c40380c86fa1"). InnerVolumeSpecName "kube-api-access-r47jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:04.439683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.439644    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.439683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.439683    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a093b4d2-d792-4541-a260-c40380c86fa1-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.439683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.439701    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r47jx\" (UniqueName: \"kubernetes.io/projected/a093b4d2-d792-4541-a260-c40380c86fa1-kube-api-access-r47jx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:04.458524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.458495    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"]
Oct 31 01:23:04.464105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.464073    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67749c6fd4-wn8q2"]
Oct 31 01:23:04.836984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:04.836948    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aed7cc5-9ca6-442d-869a-268b6c200fb0" path="/var/lib/kubelet/pods/5aed7cc5-9ca6-442d-869a-268b6c200fb0/volumes"
Oct 31 01:23:05.047156 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:05.047063    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"
Oct 31 01:23:05.047156 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:05.047100    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p" event={"ID":"a093b4d2-d792-4541-a260-c40380c86fa1","Type":"ContainerDied","Data":"ce3c9765c5ef0dc7f15db6c2243a183040248e885a271899f1fd70392d7eddca"}
Oct 31 01:23:05.047629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:05.047167    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3c9765c5ef0dc7f15db6c2243a183040248e885a271899f1fd70392d7eddca"
Oct 31 01:23:09.641325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.641274    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"]
Oct 31 01:23:09.642674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.642647    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a093b4d2-d792-4541-a260-c40380c86fa1" containerName="revoke-keys"
Oct 31 01:23:09.642818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.642803    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a093b4d2-d792-4541-a260-c40380c86fa1" containerName="revoke-keys"
Oct 31 01:23:09.642967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.642954    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5aed7cc5-9ca6-442d-869a-268b6c200fb0" containerName="payload-processing"
Oct 31 01:23:09.643051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.643039    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aed7cc5-9ca6-442d-869a-268b6c200fb0" containerName="payload-processing"
Oct 31 01:23:09.643277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.643261    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a093b4d2-d792-4541-a260-c40380c86fa1" containerName="revoke-keys"
Oct 31 01:23:09.643411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.643398    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aed7cc5-9ca6-442d-869a-268b6c200fb0" containerName="payload-processing"
Oct 31 01:23:09.647502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.647484    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.702108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.702058    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.702454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.702424    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.702675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.702653    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjfc\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.702843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.702825    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.703016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.702990    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.703202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.703184    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.703370 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.703351    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.703555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.703523    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.703818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.703786    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.764633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.764591    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"]
Oct 31 01:23:09.823319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.823277    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.837336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.837298    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.837650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.837627    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.837786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.837769    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.837975 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.837956    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjfc\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.838116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.838100    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.838282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.838264    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.838403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.838388    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.838518 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.838500    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.851301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.843324    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.851477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.847346    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.859428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.852036    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.859428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.858086    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.859428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.858838    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.875535 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.875466    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.942213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.918723    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.970164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.969942    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:09.970598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:09.970547    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjfc\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:10.269077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:10.269034    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:11.002408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:10.997668    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"]
Oct 31 01:23:11.024182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:11.023192    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:23:11.024182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:11.023276    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:23:11.024182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:11.023312    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:23:11.125786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:11.125749    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" event={"ID":"b5b6bbbf-5f94-4689-89d5-fb54ff093167","Type":"ContainerStarted","Data":"2753df029aba3f1f572956be8354aa419608a96759c487ae64c046f85019bcaf"}
Oct 31 01:23:12.172195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:12.172130    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" event={"ID":"b5b6bbbf-5f94-4689-89d5-fb54ff093167","Type":"ContainerStarted","Data":"b5981b42a6b42e68b4302d30b04e9f3ae4ac7ae779491d589981d7383db57365"}
Oct 31 01:23:12.274701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:12.274668    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:12.277370 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:12.277343    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:12.277577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:12.277551    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:12.353905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:12.353846    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podStartSLOduration=3.353792141 podStartE2EDuration="3.353792141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:12.338966997 +0000 UTC m=+1113.639923577" watchObservedRunningTime="2026-10-31 01:23:12.353792141 +0000 UTC m=+1113.654748719"
Oct 31 01:23:13.270293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:13.270256    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:13.270671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:13.270319    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:14.279174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:14.277168    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:14.279174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:14.277226    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:14.605279 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:23:14.605196    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:23:15.271032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:15.270999    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:15.271222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:15.271063    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:16.269891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:16.269852    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:16.270303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:16.269929    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:17.271849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.271811    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:17.272490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.272449    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:17.658144 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.658060    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"]
Oct 31 01:23:17.662220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.662198    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.764318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.764280    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:23:17.764529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.764290    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:23:17.764529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.764314    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:23:17.764529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.764321    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:23:17.767106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.767077    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"]
Oct 31 01:23:17.837228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.837198    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsv67\" (UniqueName: \"kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.837488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.837236    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.837488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.837400    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.938486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.938400    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.938486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.938470    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsv67\" (UniqueName: \"kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.938738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.938494    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.939162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.939118    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.940935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.940911    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:17.998599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:17.998548    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsv67\" (UniqueName: \"kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67\") pod \"maas-api-revoke-sub-7c85b3b9-78hvx\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:18.270340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:18.270311    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 01:23:18.270514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:18.270371    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 01:23:18.275030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:18.275008    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:18.659965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:18.659936    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"]
Oct 31 01:23:18.661501 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:18.661471    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f3c7c8_7fd6_4518_b408_e0208f6a6a84.slice/crio-85b4694b2c59e657066e4e6ec4b06ac4ae760e427536fde478c90f4e12f1680d WatchSource:0}: Error finding container 85b4694b2c59e657066e4e6ec4b06ac4ae760e427536fde478c90f4e12f1680d: Status 404 returned error can't find the container with id 85b4694b2c59e657066e4e6ec4b06ac4ae760e427536fde478c90f4e12f1680d
Oct 31 01:23:19.236841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:19.236809    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx" event={"ID":"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84","Type":"ContainerStarted","Data":"380ce2f8cab002df5cb982a90eb8cc0c1bcc92a878238faa526c2559bcffb92b"}
Oct 31 01:23:19.237082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:19.236868    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx" event={"ID":"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84","Type":"ContainerStarted","Data":"85b4694b2c59e657066e4e6ec4b06ac4ae760e427536fde478c90f4e12f1680d"}
Oct 31 01:23:19.354542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:19.354430    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx" podStartSLOduration=2.35441198 podStartE2EDuration="2.35441198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:19.353386125 +0000 UTC m=+1120.654342702" watchObservedRunningTime="2026-10-31 01:23:19.35441198 +0000 UTC m=+1120.655368557"
Oct 31 01:23:19.483792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:19.483753    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:19.484231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:19.484195    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:19.485661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:19.485636    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:23:20.245068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:20.245035    5419 generic.go:356] "Generic (PLEG): container finished" podID="e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" containerID="380ce2f8cab002df5cb982a90eb8cc0c1bcc92a878238faa526c2559bcffb92b" exitCode=0
Oct 31 01:23:20.245282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:20.245189    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx" event={"ID":"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84","Type":"ContainerDied","Data":"380ce2f8cab002df5cb982a90eb8cc0c1bcc92a878238faa526c2559bcffb92b"}
Oct 31 01:23:21.392122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.392099    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:21.477107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.477075    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67\" (UniqueName: \"kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67\") pod \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") "
Oct 31 01:23:21.477286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.477115    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca\") pod \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") "
Oct 31 01:23:21.477286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.477210    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca\") pod \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\" (UID: \"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84\") "
Oct 31 01:23:21.477605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.477578    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca" pod "e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" (UID: "e8f3c7c8-7fd6-4518-b408-e0208f6a6a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:21.479325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.479302    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca" pod "e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" (UID: "e8f3c7c8-7fd6-4518-b408-e0208f6a6a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:21.479410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.479338    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67" pod "e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" (UID: "e8f3c7c8-7fd6-4518-b408-e0208f6a6a84"). InnerVolumeSpecName "kube-api-access-bsv67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:21.579076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.578994    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:21.579076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.579025    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsv67\" (UniqueName: \"kubernetes.io/projected/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-kube-api-access-bsv67\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:21.579076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:21.579037    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:22.257960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:22.257930    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx" event={"ID":"e8f3c7c8-7fd6-4518-b408-e0208f6a6a84","Type":"ContainerDied","Data":"85b4694b2c59e657066e4e6ec4b06ac4ae760e427536fde478c90f4e12f1680d"}
Oct 31 01:23:22.258174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:22.257982    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b4694b2c59e657066e4e6ec4b06ac4ae760e427536fde478c90f4e12f1680d"
Oct 31 01:23:22.258174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:22.257998    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"
Oct 31 01:23:24.022764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:24.022728    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:23:25.061463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.061428    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"]
Oct 31 01:23:25.061956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.061942    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" containerName="revoke-keys"
Oct 31 01:23:25.062005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.061958    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" containerName="revoke-keys"
Oct 31 01:23:25.062045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.062036    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" containerName="revoke-keys"
Oct 31 01:23:25.065289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.065274    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.127472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.127437    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.127661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.127491    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.127661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.127558    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.127804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.127670    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.127804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.127696    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.127804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.127777    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkc8j\" (UniqueName: \"kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.167464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.167434    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"e2e-conflict-b-8bd02cd2-kserve-self-signed-certs\""
Oct 31 01:23:25.172368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.172342    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"]
Oct 31 01:23:25.229009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.228965    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.232259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.232227    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.232631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.232609    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.232769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.232754    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.232928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.232913    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkc8j\" (UniqueName: \"kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.233037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.233024    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.233954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.233927    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.234123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.230394    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.234554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.234533    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.262878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.262847    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.270533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.270498    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.304168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.304110    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkc8j\" (UniqueName: \"kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j\") pod \"e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") " pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.378341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.378246    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:25.822619 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:25.822588    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1ff6375_f75b_447b_b3f2_30184f1f94a2.slice/crio-cd4b8506dcb01264555eb5ed7ee71e2c45fa425ac65c192699b90f9c7593519f WatchSource:0}: Error finding container cd4b8506dcb01264555eb5ed7ee71e2c45fa425ac65c192699b90f9c7593519f: Status 404 returned error can't find the container with id cd4b8506dcb01264555eb5ed7ee71e2c45fa425ac65c192699b90f9c7593519f
Oct 31 01:23:25.823326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:25.823295    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"]
Oct 31 01:23:26.289459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:26.289420    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" event={"ID":"c1ff6375-f75b-447b-b3f2-30184f1f94a2","Type":"ContainerStarted","Data":"d7e178ed0e92cbe31406e9a4d888166655786f14b3b3470c887798c42dac9e37"}
Oct 31 01:23:26.289850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:26.289471    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" event={"ID":"c1ff6375-f75b-447b-b3f2-30184f1f94a2","Type":"ContainerStarted","Data":"cd4b8506dcb01264555eb5ed7ee71e2c45fa425ac65c192699b90f9c7593519f"}
Oct 31 01:23:26.379204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:26.379176    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:26.411856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:26.411798    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podStartSLOduration=1.411780134 podStartE2EDuration="1.411780134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:26.407408069 +0000 UTC m=+1127.708364648" watchObservedRunningTime="2026-10-31 01:23:26.411780134 +0000 UTC m=+1127.712736714"
Oct 31 01:23:31.816347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.816315    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"]
Oct 31 01:23:31.821736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.821712    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.840514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.840487    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.840629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.840554    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwgnr\" (UniqueName: \"kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.840629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.840618    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.877459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.877414    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"]
Oct 31 01:23:31.923462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.923418    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:23:31.923462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.923424    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:23:31.923462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.923424    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:23:31.923462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.923458    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:23:31.941588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.941559    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.941764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.941642    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwgnr\" (UniqueName: \"kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.941764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.941674    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.942189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.942173    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.944348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.944317    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:31.997860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:31.997834    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwgnr\" (UniqueName: \"kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr\") pod \"maas-api-revoke-sub-fa5bbcb6-fzzqk\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:32.139122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:32.139037    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:32.515365 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:32.515338    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"]
Oct 31 01:23:32.517523 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:32.517497    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af491e9_c2d4_437a_a697_1026757f9373.slice/crio-da6f677f92be9b0e8e29787446a7ae10e712661f9ad7364346d6cfd184539ed3 WatchSource:0}: Error finding container da6f677f92be9b0e8e29787446a7ae10e712661f9ad7364346d6cfd184539ed3: Status 404 returned error can't find the container with id da6f677f92be9b0e8e29787446a7ae10e712661f9ad7364346d6cfd184539ed3
Oct 31 01:23:33.357485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:33.357450    5419 generic.go:356] "Generic (PLEG): container finished" podID="9af491e9-c2d4-437a-a697-1026757f9373" containerID="b3779aa336a28536f622ba593784b34aab22c9111a4744cc738a42f6681ee7c6" exitCode=0
Oct 31 01:23:33.357849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:33.357547    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk" event={"ID":"9af491e9-c2d4-437a-a697-1026757f9373","Type":"ContainerDied","Data":"b3779aa336a28536f622ba593784b34aab22c9111a4744cc738a42f6681ee7c6"}
Oct 31 01:23:33.357849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:33.357584    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk" event={"ID":"9af491e9-c2d4-437a-a697-1026757f9373","Type":"ContainerStarted","Data":"da6f677f92be9b0e8e29787446a7ae10e712661f9ad7364346d6cfd184539ed3"}
Oct 31 01:23:34.507394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.507366    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:34.571235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.571207    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca\") pod \"9af491e9-c2d4-437a-a697-1026757f9373\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") "
Oct 31 01:23:34.571393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.571261    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca\") pod \"9af491e9-c2d4-437a-a697-1026757f9373\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") "
Oct 31 01:23:34.571393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.571340    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr\" (UniqueName: \"kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr\") pod \"9af491e9-c2d4-437a-a697-1026757f9373\" (UID: \"9af491e9-c2d4-437a-a697-1026757f9373\") "
Oct 31 01:23:34.571681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.571653    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca" pod "9af491e9-c2d4-437a-a697-1026757f9373" (UID: "9af491e9-c2d4-437a-a697-1026757f9373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:34.573355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.573331    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca" pod "9af491e9-c2d4-437a-a697-1026757f9373" (UID: "9af491e9-c2d4-437a-a697-1026757f9373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:34.573561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.573536    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr" pod "9af491e9-c2d4-437a-a697-1026757f9373" (UID: "9af491e9-c2d4-437a-a697-1026757f9373"). InnerVolumeSpecName "kube-api-access-bwgnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:34.672546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.672458    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:34.672546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.672490    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9af491e9-c2d4-437a-a697-1026757f9373-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:34.672546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:34.672504    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwgnr\" (UniqueName: \"kubernetes.io/projected/9af491e9-c2d4-437a-a697-1026757f9373-kube-api-access-bwgnr\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:35.370191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:35.370155    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk" event={"ID":"9af491e9-c2d4-437a-a697-1026757f9373","Type":"ContainerDied","Data":"da6f677f92be9b0e8e29787446a7ae10e712661f9ad7364346d6cfd184539ed3"}
Oct 31 01:23:35.370191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:35.370195    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"
Oct 31 01:23:35.370382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:35.370216    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6f677f92be9b0e8e29787446a7ae10e712661f9ad7364346d6cfd184539ed3"
Oct 31 01:23:35.378685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:35.378654    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:35.391865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:35.391841    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:36.029953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.029918    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"]
Oct 31 01:23:36.030424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.030411    5419 cpu_manager.go:414] "containerMap: removing container" podUID="9af491e9-c2d4-437a-a697-1026757f9373" containerName="revoke-keys"
Oct 31 01:23:36.030468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.030425    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af491e9-c2d4-437a-a697-1026757f9373" containerName="revoke-keys"
Oct 31 01:23:36.030522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.030512    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af491e9-c2d4-437a-a697-1026757f9373" containerName="revoke-keys"
Oct 31 01:23:36.033895 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.033880    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.088758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.088723    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.088913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.088766    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.088913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.088846    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.088913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.088894    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.089035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.088919    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gmp\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.089069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.089040    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.089069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.089057    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.089159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.089079    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.089222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.089202    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.137782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.137747    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"]
Oct 31 01:23:36.189980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.189947    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.189989    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gmp\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190053    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190070    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190103    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190180    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190240    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190273    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190318    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190522    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190603    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190662    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.190803 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190785    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.191025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.190907    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.192625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.192605    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.192728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.192694    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.246111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.246076    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.246281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.246235    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gmp\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.347263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.347184    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:36.391237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.391203    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:23:36.776792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.776766    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"]
Oct 31 01:23:36.780826 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:36.780796    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd493a02_5f4e_4680_b77f_f4f29830a0e3.slice/crio-e8c11b78784879b03afec22f2cc0c02421f2ed757bf14366b9b1e1882e34c16f WatchSource:0}: Error finding container e8c11b78784879b03afec22f2cc0c02421f2ed757bf14366b9b1e1882e34c16f: Status 404 returned error can't find the container with id e8c11b78784879b03afec22f2cc0c02421f2ed757bf14366b9b1e1882e34c16f
Oct 31 01:23:36.783721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.783674    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:23:36.783847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.783761    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:23:36.783847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:36.783805    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:23:37.405164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:37.404421    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" event={"ID":"dd493a02-5f4e-4680-b77f-f4f29830a0e3","Type":"ContainerStarted","Data":"31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721"}
Oct 31 01:23:37.405164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:37.404497    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" event={"ID":"dd493a02-5f4e-4680-b77f-f4f29830a0e3","Type":"ContainerStarted","Data":"e8c11b78784879b03afec22f2cc0c02421f2ed757bf14366b9b1e1882e34c16f"}
Oct 31 01:23:37.522133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:37.522078    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podStartSLOduration=2.522064317 podStartE2EDuration="2.522064317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:37.520360096 +0000 UTC m=+1138.821316694" watchObservedRunningTime="2026-10-31 01:23:37.522064317 +0000 UTC m=+1138.823020893"
Oct 31 01:23:38.347874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:38.347836    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:38.349549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:38.349525    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:38.349666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:38.349573    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:39.348440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:39.348409    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:39.348850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:39.348462    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:40.348433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.348397    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:40.348616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.348454    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:40.751342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.751310    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"]
Oct 31 01:23:40.756476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.756455    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.812421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.812386    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"]
Oct 31 01:23:40.857755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.857725    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:23:40.857902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.857767    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:23:40.857902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.857767    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:23:40.857902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.857767    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:23:40.859429 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.859412    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.859478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.859454    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.859535 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.859480    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhrgd\" (UniqueName: \"kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.960656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.960626    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.960848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.960670    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhrgd\" (UniqueName: \"kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.960848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.960794    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.961306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.961277    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:40.963313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:40.963288    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:41.017992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:41.017924    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhrgd\" (UniqueName: \"kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd\") pod \"maas-api-revoke-sub-5b6c9ea7-4lb79\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:41.069943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:41.069910    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:41.347652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:41.347566    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:41.347652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:41.347626    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:41.468671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:41.468630    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"]
Oct 31 01:23:41.469410 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:41.469382    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf22a86_37a0_48c1_977d_1bea32f27b2a.slice/crio-88b22070366e05a07bd81912b0c22438bf84e1a962c33493ade9e83274eb5803 WatchSource:0}: Error finding container 88b22070366e05a07bd81912b0c22438bf84e1a962c33493ade9e83274eb5803: Status 404 returned error can't find the container with id 88b22070366e05a07bd81912b0c22438bf84e1a962c33493ade9e83274eb5803
Oct 31 01:23:42.051833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.051795    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"]
Oct 31 01:23:42.054627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.054597    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1483c614-2mplw"]
Oct 31 01:23:42.347671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.347594    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:42.347671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.347647    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:42.435946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.435904    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79" event={"ID":"eaf22a86-37a0-48c1-977d-1bea32f27b2a","Type":"ContainerStarted","Data":"89316f23e308ff008a78fa5be6cdcc17889ebe68995d10fe70b6f6d9c8f9625a"}
Oct 31 01:23:42.436116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.435956    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79" event={"ID":"eaf22a86-37a0-48c1-977d-1bea32f27b2a","Type":"ContainerStarted","Data":"88b22070366e05a07bd81912b0c22438bf84e1a962c33493ade9e83274eb5803"}
Oct 31 01:23:42.550704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.550654    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79" podStartSLOduration=2.550640348 podStartE2EDuration="2.550640348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:42.547994489 +0000 UTC m=+1143.848951065" watchObservedRunningTime="2026-10-31 01:23:42.550640348 +0000 UTC m=+1143.851596925"
Oct 31 01:23:42.604282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.604213    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"]
Oct 31 01:23:42.608282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.608260    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.656624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.656595    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"]
Oct 31 01:23:42.703888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.703844    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvgn\" (UniqueName: \"kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.704088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.703905    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.704222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.704204    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.805756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.805723    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.805957 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.805794    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvgn\" (UniqueName: \"kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.805957 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.805840    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.806762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.806729    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.808308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.808286    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.841188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.841160    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06" path="/var/lib/kubelet/pods/c34375ce-4ccf-4ba4-88be-5ca1b1b7fb06/volumes"
Oct 31 01:23:42.862492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.862432    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvgn\" (UniqueName: \"kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn\") pod \"maas-api-revoke-sub-b53e9b01-nxwxt\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:42.950614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:42.950581    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:43.324308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.324284    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"]
Oct 31 01:23:43.326644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:43.326617    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d2da4f_c02d_41e7_8d63_48b82642881a.slice/crio-75339c8bf0962d3ec820a213d746c6459ee91916bf3e2be30e03fcda1b0cbc6a WatchSource:0}: Error finding container 75339c8bf0962d3ec820a213d746c6459ee91916bf3e2be30e03fcda1b0cbc6a: Status 404 returned error can't find the container with id 75339c8bf0962d3ec820a213d746c6459ee91916bf3e2be30e03fcda1b0cbc6a
Oct 31 01:23:43.348712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.348684    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 31 01:23:43.348845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.348741    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 31 01:23:43.453162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.452775    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt" event={"ID":"95d2da4f-c02d-41e7-8d63-48b82642881a","Type":"ContainerStarted","Data":"75339c8bf0962d3ec820a213d746c6459ee91916bf3e2be30e03fcda1b0cbc6a"}
Oct 31 01:23:43.467731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.463280    5419 generic.go:356] "Generic (PLEG): container finished" podID="eaf22a86-37a0-48c1-977d-1bea32f27b2a" containerID="89316f23e308ff008a78fa5be6cdcc17889ebe68995d10fe70b6f6d9c8f9625a" exitCode=0
Oct 31 01:23:43.467731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.463389    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79" event={"ID":"eaf22a86-37a0-48c1-977d-1bea32f27b2a","Type":"ContainerDied","Data":"89316f23e308ff008a78fa5be6cdcc17889ebe68995d10fe70b6f6d9c8f9625a"}
Oct 31 01:23:43.548018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.547984    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"]
Oct 31 01:23:43.553104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.553076    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.610013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.609983    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"]
Oct 31 01:23:43.715952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.715903    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8q4\" (UniqueName: \"kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.716159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.716043    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.716259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.716178    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.817166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.817111    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.817344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.817239    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.817437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.817339    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8q4\" (UniqueName: \"kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.817858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.817830    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.820124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.820099    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:43.874248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:43.874166    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8q4\" (UniqueName: \"kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4\") pod \"maas-api-revoke-sub-43893bca-hmckf\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:44.168571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:44.168468    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:44.480829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:44.480787    5419 generic.go:356] "Generic (PLEG): container finished" podID="95d2da4f-c02d-41e7-8d63-48b82642881a" containerID="38454dc3d9f053b11ba594812650bd11a35f379a750ec8910a251b75d157c46b" exitCode=0
Oct 31 01:23:44.481558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:44.481532    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt" event={"ID":"95d2da4f-c02d-41e7-8d63-48b82642881a","Type":"ContainerDied","Data":"38454dc3d9f053b11ba594812650bd11a35f379a750ec8910a251b75d157c46b"}
Oct 31 01:23:44.576166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:44.571200    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"]
Oct 31 01:23:44.993186 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:23:44.974188    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:23:45.354857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.352914    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:45.401346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.401285    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:23:45.401695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.401666    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:23:45.534243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.528199    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd\" (UniqueName: \"kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd\") pod \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") "
Oct 31 01:23:45.534243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.528281    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca\") pod \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") "
Oct 31 01:23:45.534243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.528419    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca\") pod \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\" (UID: \"eaf22a86-37a0-48c1-977d-1bea32f27b2a\") "
Oct 31 01:23:45.566195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.566156    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca" pod "eaf22a86-37a0-48c1-977d-1bea32f27b2a" (UID: "eaf22a86-37a0-48c1-977d-1bea32f27b2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:45.653303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.651594    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:45.725239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.719255    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca" pod "eaf22a86-37a0-48c1-977d-1bea32f27b2a" (UID: "eaf22a86-37a0-48c1-977d-1bea32f27b2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:45.743458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.743422    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf" event={"ID":"2fe463af-30de-445f-8c84-464992ae3049","Type":"ContainerStarted","Data":"6ed51455afa4357a49777743a0b474304c779e884da8bec4a73602c26452f0a5"}
Oct 31 01:23:45.746521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.746501    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"
Oct 31 01:23:45.746934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.746889    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79" event={"ID":"eaf22a86-37a0-48c1-977d-1bea32f27b2a","Type":"ContainerDied","Data":"88b22070366e05a07bd81912b0c22438bf84e1a962c33493ade9e83274eb5803"}
Oct 31 01:23:45.747108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.747091    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88b22070366e05a07bd81912b0c22438bf84e1a962c33493ade9e83274eb5803"
Oct 31 01:23:45.753225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.753199    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf22a86-37a0-48c1-977d-1bea32f27b2a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:45.829078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.827282    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd" pod "eaf22a86-37a0-48c1-977d-1bea32f27b2a" (UID: "eaf22a86-37a0-48c1-977d-1bea32f27b2a"). InnerVolumeSpecName "kube-api-access-nhrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:45.875088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:45.874970    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhrgd\" (UniqueName: \"kubernetes.io/projected/eaf22a86-37a0-48c1-977d-1bea32f27b2a-kube-api-access-nhrgd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:46.389791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.388461    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded"
Oct 31 01:23:46.389791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.389215    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:23:46.452715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.452677    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:23:46.453011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.452982    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:23:46.558216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.558187    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:46.640212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.638425    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:46.640212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.639619    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:46.640970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.640913    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:23:46.743743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.743700    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca\") pod \"95d2da4f-c02d-41e7-8d63-48b82642881a\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") "
Oct 31 01:23:46.744257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.743769    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca\") pod \"95d2da4f-c02d-41e7-8d63-48b82642881a\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") "
Oct 31 01:23:46.744257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.743948    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn\" (UniqueName: \"kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn\") pod \"95d2da4f-c02d-41e7-8d63-48b82642881a\" (UID: \"95d2da4f-c02d-41e7-8d63-48b82642881a\") "
Oct 31 01:23:46.744257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.744121    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca" pod "95d2da4f-c02d-41e7-8d63-48b82642881a" (UID: "95d2da4f-c02d-41e7-8d63-48b82642881a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:46.744423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.744409    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:46.746025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.745997    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca" pod "95d2da4f-c02d-41e7-8d63-48b82642881a" (UID: "95d2da4f-c02d-41e7-8d63-48b82642881a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:46.746570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.746541    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn" pod "95d2da4f-c02d-41e7-8d63-48b82642881a" (UID: "95d2da4f-c02d-41e7-8d63-48b82642881a"). InnerVolumeSpecName "kube-api-access-ccvgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:46.754214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.754182    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt" event={"ID":"95d2da4f-c02d-41e7-8d63-48b82642881a","Type":"ContainerDied","Data":"75339c8bf0962d3ec820a213d746c6459ee91916bf3e2be30e03fcda1b0cbc6a"}
Oct 31 01:23:46.754305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.754214    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"
Oct 31 01:23:46.754305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.754249    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75339c8bf0962d3ec820a213d746c6459ee91916bf3e2be30e03fcda1b0cbc6a"
Oct 31 01:23:46.756082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.756055    5419 generic.go:356] "Generic (PLEG): container finished" podID="2fe463af-30de-445f-8c84-464992ae3049" containerID="d9bded28056bd91ebb6c193f04d0e71510fb42d2db98372072804d282c4d90bf" exitCode=0
Oct 31 01:23:46.756235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.756088    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf" event={"ID":"2fe463af-30de-445f-8c84-464992ae3049","Type":"ContainerDied","Data":"d9bded28056bd91ebb6c193f04d0e71510fb42d2db98372072804d282c4d90bf"}
Oct 31 01:23:46.846353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.846324    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95d2da4f-c02d-41e7-8d63-48b82642881a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:46.846596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:46.846568    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvgn\" (UniqueName: \"kubernetes.io/projected/95d2da4f-c02d-41e7-8d63-48b82642881a-kube-api-access-ccvgn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:47.904339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.904312    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:47.956091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.956065    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca\") pod \"2fe463af-30de-445f-8c84-464992ae3049\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") "
Oct 31 01:23:47.956244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.956108    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4\" (UniqueName: \"kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4\") pod \"2fe463af-30de-445f-8c84-464992ae3049\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") "
Oct 31 01:23:47.956244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.956198    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca\") pod \"2fe463af-30de-445f-8c84-464992ae3049\" (UID: \"2fe463af-30de-445f-8c84-464992ae3049\") "
Oct 31 01:23:47.956593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.956568    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca" pod "2fe463af-30de-445f-8c84-464992ae3049" (UID: "2fe463af-30de-445f-8c84-464992ae3049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:47.958357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.958334    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca" pod "2fe463af-30de-445f-8c84-464992ae3049" (UID: "2fe463af-30de-445f-8c84-464992ae3049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:47.958489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:47.958471    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4" pod "2fe463af-30de-445f-8c84-464992ae3049" (UID: "2fe463af-30de-445f-8c84-464992ae3049"). InnerVolumeSpecName "kube-api-access-6x8q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:48.057632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:48.057558    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8q4\" (UniqueName: \"kubernetes.io/projected/2fe463af-30de-445f-8c84-464992ae3049-kube-api-access-6x8q4\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:48.057632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:48.057588    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe463af-30de-445f-8c84-464992ae3049-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:48.057632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:48.057600    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe463af-30de-445f-8c84-464992ae3049-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:48.770033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:48.770001    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"
Oct 31 01:23:48.770225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:48.770027    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf" event={"ID":"2fe463af-30de-445f-8c84-464992ae3049","Type":"ContainerDied","Data":"6ed51455afa4357a49777743a0b474304c779e884da8bec4a73602c26452f0a5"}
Oct 31 01:23:48.770225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:48.770078    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed51455afa4357a49777743a0b474304c779e884da8bec4a73602c26452f0a5"
Oct 31 01:23:54.372032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.371993    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"]
Oct 31 01:23:54.372548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372490    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2fe463af-30de-445f-8c84-464992ae3049" containerName="revoke-keys"
Oct 31 01:23:54.372548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372501    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe463af-30de-445f-8c84-464992ae3049" containerName="revoke-keys"
Oct 31 01:23:54.372548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372524    5419 cpu_manager.go:414] "containerMap: removing container" podUID="95d2da4f-c02d-41e7-8d63-48b82642881a" containerName="revoke-keys"
Oct 31 01:23:54.372548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372528    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d2da4f-c02d-41e7-8d63-48b82642881a" containerName="revoke-keys"
Oct 31 01:23:54.372548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372547    5419 cpu_manager.go:414] "containerMap: removing container" podUID="eaf22a86-37a0-48c1-977d-1bea32f27b2a" containerName="revoke-keys"
Oct 31 01:23:54.372711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372553    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf22a86-37a0-48c1-977d-1bea32f27b2a" containerName="revoke-keys"
Oct 31 01:23:54.372711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372633    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe463af-30de-445f-8c84-464992ae3049" containerName="revoke-keys"
Oct 31 01:23:54.372711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372642    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d2da4f-c02d-41e7-8d63-48b82642881a" containerName="revoke-keys"
Oct 31 01:23:54.372711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.372650    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf22a86-37a0-48c1-977d-1bea32f27b2a" containerName="revoke-keys"
Oct 31 01:23:54.376111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.376093    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.430603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.430567    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 01:23:54.431024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.431002    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 01:23:54.458538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.458494    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.458895 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.458872    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.459108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.459091    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.459454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.459434    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjj9\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.479920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.479894    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1984e9af4453\""
Oct 31 01:23:54.480174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.480123    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 01:23:54.480300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.480174    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 01:23:54.480300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.480208    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-7qstt\""
Oct 31 01:23:54.483258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.483240    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"]
Oct 31 01:23:54.561039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.561015    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjj9\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.561259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.561050    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.561259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.561073    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.561259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.561096    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.561694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.561668    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.561844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.561822    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.563648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.563632    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.622811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.622743    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjj9\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9\") pod \"payload-processing-external-model-5dcfb47976-czmqt\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") " pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.690463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.690436    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:54.837965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.837939    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"]
Oct 31 01:23:54.841792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.841777    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.864099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.864070    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w4wq\" (UniqueName: \"kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.864247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.864168    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.864247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.864231    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.915214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.906346    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:23:54.915214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.911319    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:23:54.915214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.911485    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:23:54.915214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.911582    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:23:54.966182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.948800    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"]
Oct 31 01:23:54.987167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.974668    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w4wq\" (UniqueName: \"kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.987167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.974773    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.987167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.974816    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:54.987167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:54.976568    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:55.025681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.025632    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:55.031646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.031609    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w4wq\" (UniqueName: \"kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq\") pod \"maas-api-revoke-sub-add3ec25-xj65h\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:55.092901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.092863    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"]
Oct 31 01:23:55.093949 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:55.093925    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5229b736_3aa1_4218_8817_ce7448d18447.slice/crio-bce699b4b63e10d75d898031b3e1f9b1e61bce9e7522923907dda58011362bf1 WatchSource:0}: Error finding container bce699b4b63e10d75d898031b3e1f9b1e61bce9e7522923907dda58011362bf1: Status 404 returned error can't find the container with id bce699b4b63e10d75d898031b3e1f9b1e61bce9e7522923907dda58011362bf1
Oct 31 01:23:55.155558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.155535    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:55.536257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.536231    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"]
Oct 31 01:23:55.536703 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:23:55.536680    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d6c7d2_e25d_4d5f_818b_2cac7c35b18f.slice/crio-ad9bfd81f255fa0c8140dc91290549812855d525ec4c989c81b1a2b5797fa631 WatchSource:0}: Error finding container ad9bfd81f255fa0c8140dc91290549812855d525ec4c989c81b1a2b5797fa631: Status 404 returned error can't find the container with id ad9bfd81f255fa0c8140dc91290549812855d525ec4c989c81b1a2b5797fa631
Oct 31 01:23:55.864709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.864680    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h" event={"ID":"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f","Type":"ContainerStarted","Data":"a27f4f273d4c3149d3b7dac4f574a566568b541b0ad6993db754e95197172f97"}
Oct 31 01:23:55.864889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.864730    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h" event={"ID":"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f","Type":"ContainerStarted","Data":"ad9bfd81f255fa0c8140dc91290549812855d525ec4c989c81b1a2b5797fa631"}
Oct 31 01:23:55.866602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.866573    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" event={"ID":"5229b736-3aa1-4218-8817-ce7448d18447","Type":"ContainerStarted","Data":"ba76bcf4eeac33e2d56614e86de09d1d0f2b97fc9f5afd67338a7a363210d8f7"}
Oct 31 01:23:55.866736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.866624    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" event={"ID":"5229b736-3aa1-4218-8817-ce7448d18447","Type":"ContainerStarted","Data":"bce699b4b63e10d75d898031b3e1f9b1e61bce9e7522923907dda58011362bf1"}
Oct 31 01:23:55.866736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.866663    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:23:55.980299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:55.980242    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h" podStartSLOduration=1.980227761 podStartE2EDuration="1.980227761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:55.9782248 +0000 UTC m=+1157.279181380" watchObservedRunningTime="2026-10-31 01:23:55.980227761 +0000 UTC m=+1157.281184354"
Oct 31 01:23:56.094604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:56.094492    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" podStartSLOduration=2.09447675 podStartE2EDuration="2.09447675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:56.090898728 +0000 UTC m=+1157.391855318" watchObservedRunningTime="2026-10-31 01:23:56.09447675 +0000 UTC m=+1157.395433373"
Oct 31 01:23:56.873939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:56.873899    5419 generic.go:356] "Generic (PLEG): container finished" podID="b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" containerID="a27f4f273d4c3149d3b7dac4f574a566568b541b0ad6993db754e95197172f97" exitCode=0
Oct 31 01:23:56.874338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:56.873982    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h" event={"ID":"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f","Type":"ContainerDied","Data":"a27f4f273d4c3149d3b7dac4f574a566568b541b0ad6993db754e95197172f97"}
Oct 31 01:23:58.036959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.036930    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:58.118668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.118643    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca\") pod \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") "
Oct 31 01:23:58.118875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.118857    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq\" (UniqueName: \"kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq\") pod \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") "
Oct 31 01:23:58.118940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.118903    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca\") pod \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\" (UID: \"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f\") "
Oct 31 01:23:58.119053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.119032    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca" pod "b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" (UID: "b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:58.119306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.119285    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:58.121460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.121430    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq" pod "b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" (UID: "b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f"). InnerVolumeSpecName "kube-api-access-8w4wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:58.121592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.121573    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca" pod "b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" (UID: "b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:58.220788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.220755    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w4wq\" (UniqueName: \"kubernetes.io/projected/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-kube-api-access-8w4wq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:58.220788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.220789    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:23:58.887396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.887369    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"
Oct 31 01:23:58.887563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.887445    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h" event={"ID":"b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f","Type":"ContainerDied","Data":"ad9bfd81f255fa0c8140dc91290549812855d525ec4c989c81b1a2b5797fa631"}
Oct 31 01:23:58.887563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:58.887494    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9bfd81f255fa0c8140dc91290549812855d525ec4c989c81b1a2b5797fa631"
Oct 31 01:23:59.107937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.107849    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:23:59.413414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.413322    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"]
Oct 31 01:23:59.413933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.413913    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" containerName="revoke-keys"
Oct 31 01:23:59.414036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.413936    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" containerName="revoke-keys"
Oct 31 01:23:59.414090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.414063    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" containerName="revoke-keys"
Oct 31 01:23:59.417497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.417472    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.435544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.435514    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.435675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.435642    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.435749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.435733    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lr95\" (UniqueName: \"kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.473384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.473349    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:23:59.473655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.473625    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:23:59.473909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.473890    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:23:59.579243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.579210    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:23:59.599002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.598216    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"]
Oct 31 01:23:59.603276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.603246    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.603543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.603525    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.603686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.603672    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lr95\" (UniqueName: \"kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.604227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.604125    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.607263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.607240    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.664808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.664734    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lr95\" (UniqueName: \"kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95\") pod \"maas-api-revoke-sub-9f98a027-22qzp\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:23:59.732313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:23:59.732274    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:24:00.034293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.034260    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"]
Oct 31 01:24:00.044669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.044641    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.087878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.087215    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"]
Oct 31 01:24:00.092166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.088102    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-052d2b3f-mmrz7"]
Oct 31 01:24:00.176244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.176207    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"]
Oct 31 01:24:00.252935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.252894    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqwb\" (UniqueName: \"kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.260630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.260593    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.260900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.260857    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.398206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.394473    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"]
Oct 31 01:24:00.399114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.399082    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqwb\" (UniqueName: \"kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.407262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.407232    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.407537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.407512    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.408982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.408959    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.466889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.466857    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.506189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.503054    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqwb\" (UniqueName: \"kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb\") pod \"maas-api-revoke-sub-5c7fbb2f-q75jn\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.757759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.756310    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:00.905165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:00.893584    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33362e87-d320-46ac-8cc2-2e8d767e1f3a" path="/var/lib/kubelet/pods/33362e87-d320-46ac-8cc2-2e8d767e1f3a/volumes"
Oct 31 01:24:01.026167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:01.025002    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp" event={"ID":"fd741700-44e7-4767-ba82-cb78ffe52349","Type":"ContainerStarted","Data":"6d325dddda147afee642193eb52d35166a4f4dde62e7c87ce77e90b418e151a5"}
Oct 31 01:24:01.134164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:01.131292    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:01.326925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:01.326248    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:01.406410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:01.406366    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:01.476163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:01.447234    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:02.065165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.063207    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": context deadline exceeded"
Oct 31 01:24:02.065165 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:02.063507    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5830bedb_a59e_4144_855d_92e5a6fa2e09.slice/crio-8eb62ad5ad4c1a3b40efdd7c8e636d31ecfd0cc16c5b2b928c33e80ca63ad7f2 WatchSource:0}: Error finding container 8eb62ad5ad4c1a3b40efdd7c8e636d31ecfd0cc16c5b2b928c33e80ca63ad7f2: Status 404 returned error can't find the container with id 8eb62ad5ad4c1a3b40efdd7c8e636d31ecfd0cc16c5b2b928c33e80ca63ad7f2
Oct 31 01:24:02.085162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.083308    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"]
Oct 31 01:24:02.146174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.135221    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 01:24:02.146174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.136964    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:24:02.160372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.160332    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp" event={"ID":"fd741700-44e7-4767-ba82-cb78ffe52349","Type":"ContainerStarted","Data":"1dd07cbf9b11c917b2bce6b9caf58e4a7ddc53588ee9b1a7ae4c68c9d40cceeb"}
Oct 31 01:24:02.394817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.355243    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:02.394817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.392158    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:24:02.430008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.424752    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:02.430008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.426714    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:24:02.487336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.487282    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:02.620615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.620534    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:24:02.638589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.638565    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:24:02.639558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.639537    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:24:02.659388 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:24:02.659352    5419 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd741700_44e7_4767_ba82_cb78ffe52349.slice/crio-1dd07cbf9b11c917b2bce6b9caf58e4a7ddc53588ee9b1a7ae4c68c9d40cceeb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd741700_44e7_4767_ba82_cb78ffe52349.slice/crio-conmon-1dd07cbf9b11c917b2bce6b9caf58e4a7ddc53588ee9b1a7ae4c68c9d40cceeb.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:24:02.921155 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.921056    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 01:24:02.950448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.946765    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp" podStartSLOduration=3.946744552 podStartE2EDuration="3.946744552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:02.915474445 +0000 UTC m=+1164.216431025" watchObservedRunningTime="2026-10-31 01:24:02.946744552 +0000 UTC m=+1164.247701129"
Oct 31 01:24:02.956015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.955977    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:02.969166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:02.966233    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:24:03.168168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:03.168103    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn" event={"ID":"5830bedb-a59e-4144-855d-92e5a6fa2e09","Type":"ContainerStarted","Data":"7a9993cf25575dac08ad7ceb3fa3fb890b6cadbafe4d27ecacbd86c6ca6bc9c7"}
Oct 31 01:24:03.168331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:03.168186    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn" event={"ID":"5830bedb-a59e-4144-855d-92e5a6fa2e09","Type":"ContainerStarted","Data":"8eb62ad5ad4c1a3b40efdd7c8e636d31ecfd0cc16c5b2b928c33e80ca63ad7f2"}
Oct 31 01:24:03.170076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:03.170038    5419 generic.go:356] "Generic (PLEG): container finished" podID="fd741700-44e7-4767-ba82-cb78ffe52349" containerID="1dd07cbf9b11c917b2bce6b9caf58e4a7ddc53588ee9b1a7ae4c68c9d40cceeb" exitCode=0
Oct 31 01:24:03.170226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:03.170169    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp" event={"ID":"fd741700-44e7-4767-ba82-cb78ffe52349","Type":"ContainerDied","Data":"1dd07cbf9b11c917b2bce6b9caf58e4a7ddc53588ee9b1a7ae4c68c9d40cceeb"}
Oct 31 01:24:03.801123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:03.801069    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn" podStartSLOduration=4.801051456 podStartE2EDuration="4.801051456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:03.7984033 +0000 UTC m=+1165.099359877" watchObservedRunningTime="2026-10-31 01:24:03.801051456 +0000 UTC m=+1165.102008050"
Oct 31 01:24:04.178424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.178391    5419 generic.go:356] "Generic (PLEG): container finished" podID="5830bedb-a59e-4144-855d-92e5a6fa2e09" containerID="7a9993cf25575dac08ad7ceb3fa3fb890b6cadbafe4d27ecacbd86c6ca6bc9c7" exitCode=0
Oct 31 01:24:04.178605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.178476    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn" event={"ID":"5830bedb-a59e-4144-855d-92e5a6fa2e09","Type":"ContainerDied","Data":"7a9993cf25575dac08ad7ceb3fa3fb890b6cadbafe4d27ecacbd86c6ca6bc9c7"}
Oct 31 01:24:04.314557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.314531    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:24:04.370069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.370041    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca\") pod \"fd741700-44e7-4767-ba82-cb78ffe52349\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") "
Oct 31 01:24:04.370264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.370088    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95\" (UniqueName: \"kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95\") pod \"fd741700-44e7-4767-ba82-cb78ffe52349\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") "
Oct 31 01:24:04.370264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.370173    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca\") pod \"fd741700-44e7-4767-ba82-cb78ffe52349\" (UID: \"fd741700-44e7-4767-ba82-cb78ffe52349\") "
Oct 31 01:24:04.370406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.370378    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca" pod "fd741700-44e7-4767-ba82-cb78ffe52349" (UID: "fd741700-44e7-4767-ba82-cb78ffe52349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:04.370594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.370576    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:04.372269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.372248    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca" pod "fd741700-44e7-4767-ba82-cb78ffe52349" (UID: "fd741700-44e7-4767-ba82-cb78ffe52349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:04.372338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.372317    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95" pod "fd741700-44e7-4767-ba82-cb78ffe52349" (UID: "fd741700-44e7-4767-ba82-cb78ffe52349"). InnerVolumeSpecName "kube-api-access-8lr95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:04.471537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.471504    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lr95\" (UniqueName: \"kubernetes.io/projected/fd741700-44e7-4767-ba82-cb78ffe52349-kube-api-access-8lr95\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:04.471537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:04.471536    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd741700-44e7-4767-ba82-cb78ffe52349-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:05.185871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.185786    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"
Oct 31 01:24:05.185871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.185815    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp" event={"ID":"fd741700-44e7-4767-ba82-cb78ffe52349","Type":"ContainerDied","Data":"6d325dddda147afee642193eb52d35166a4f4dde62e7c87ce77e90b418e151a5"}
Oct 31 01:24:05.185871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.185861    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d325dddda147afee642193eb52d35166a4f4dde62e7c87ce77e90b418e151a5"
Oct 31 01:24:05.337361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.337328    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:05.381045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.381005    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca\") pod \"5830bedb-a59e-4144-855d-92e5a6fa2e09\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") "
Oct 31 01:24:05.381235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.381072    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca\") pod \"5830bedb-a59e-4144-855d-92e5a6fa2e09\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") "
Oct 31 01:24:05.381235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.381210    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb\" (UniqueName: \"kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb\") pod \"5830bedb-a59e-4144-855d-92e5a6fa2e09\" (UID: \"5830bedb-a59e-4144-855d-92e5a6fa2e09\") "
Oct 31 01:24:05.381415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.381393    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca" pod "5830bedb-a59e-4144-855d-92e5a6fa2e09" (UID: "5830bedb-a59e-4144-855d-92e5a6fa2e09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:05.381609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.381593    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:05.383344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.383324    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca" pod "5830bedb-a59e-4144-855d-92e5a6fa2e09" (UID: "5830bedb-a59e-4144-855d-92e5a6fa2e09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:05.383433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.383414    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb" pod "5830bedb-a59e-4144-855d-92e5a6fa2e09" (UID: "5830bedb-a59e-4144-855d-92e5a6fa2e09"). InnerVolumeSpecName "kube-api-access-tmqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:05.482907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.482871    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5830bedb-a59e-4144-855d-92e5a6fa2e09-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:05.482907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:05.482901    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqwb\" (UniqueName: \"kubernetes.io/projected/5830bedb-a59e-4144-855d-92e5a6fa2e09-kube-api-access-tmqwb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:06.192971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:06.192939    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"
Oct 31 01:24:06.193413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:06.192971    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn" event={"ID":"5830bedb-a59e-4144-855d-92e5a6fa2e09","Type":"ContainerDied","Data":"8eb62ad5ad4c1a3b40efdd7c8e636d31ecfd0cc16c5b2b928c33e80ca63ad7f2"}
Oct 31 01:24:06.193413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:06.193016    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eb62ad5ad4c1a3b40efdd7c8e636d31ecfd0cc16c5b2b928c33e80ca63ad7f2"
Oct 31 01:24:06.877431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:06.877395    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" podUID="5229b736-3aa1-4218-8817-ce7448d18447" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:07.101867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:07.101834    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"]
Oct 31 01:24:07.103027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:07.103001    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" podUID="5229b736-3aa1-4218-8817-ce7448d18447" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:24:10.117038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117005    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"]
Oct 31 01:24:10.117539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117525    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fd741700-44e7-4767-ba82-cb78ffe52349" containerName="revoke-keys"
Oct 31 01:24:10.117579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117540    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd741700-44e7-4767-ba82-cb78ffe52349" containerName="revoke-keys"
Oct 31 01:24:10.117579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117559    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5830bedb-a59e-4144-855d-92e5a6fa2e09" containerName="revoke-keys"
Oct 31 01:24:10.117579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117565    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5830bedb-a59e-4144-855d-92e5a6fa2e09" containerName="revoke-keys"
Oct 31 01:24:10.117670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117659    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd741700-44e7-4767-ba82-cb78ffe52349" containerName="revoke-keys"
Oct 31 01:24:10.117670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.117669    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5830bedb-a59e-4144-855d-92e5a6fa2e09" containerName="revoke-keys"
Oct 31 01:24:10.121250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.121228    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162523    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162561    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162651    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162739    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162792    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162818    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162907    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162949    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k792g\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.162995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.162975    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.222359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.222331    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"]
Oct 31 01:24:10.264565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264529    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.264724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264605    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.264724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264623    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k792g\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.264724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264647    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.264724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264682    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.264724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264701    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264761    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264824    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.264871    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.265077    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.265227    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.265260    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.265300    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.265462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.265444    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.267186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.267161    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.267595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.267576    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.321106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.321079    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.321346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.321323    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k792g\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.434973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.434898    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:10.787642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.787612    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"]
Oct 31 01:24:10.792920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.792905    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.845553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.845519    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:24:10.845553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.845528    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:24:10.845779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.845563    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:24:10.845779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.845592    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:24:10.847846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.847824    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"]
Oct 31 01:24:10.871078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.871041    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzp8p\" (UniqueName: \"kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.871259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.871240    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.871322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.871282    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.915352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.915326    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"]
Oct 31 01:24:10.916227 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:10.916204    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb32e360_3c70_43bd_a19a_86d25a2d0d3e.slice/crio-da55b19e58e0aae10091706e1e0c26e76f8d38a8f9d5f49c4d9c10361cd297c8 WatchSource:0}: Error finding container da55b19e58e0aae10091706e1e0c26e76f8d38a8f9d5f49c4d9c10361cd297c8: Status 404 returned error can't find the container with id da55b19e58e0aae10091706e1e0c26e76f8d38a8f9d5f49c4d9c10361cd297c8
Oct 31 01:24:10.918636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.918607    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:10.918707 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.918670    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:10.918707 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.918697    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:10.972670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.972633    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.972768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.972682    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.972832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.972783    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzp8p\" (UniqueName: \"kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.973348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.973324    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:10.975269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:10.975248    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:11.029045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.029015    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzp8p\" (UniqueName: \"kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p\") pod \"maas-api-revoke-sub-dbc69b53-gbc4h\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:11.043913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.043843    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"]
Oct 31 01:24:11.048313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.048293    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.106405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.106374    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:11.107557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.107531    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"]
Oct 31 01:24:11.176334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176305    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176370    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176402    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwkkp\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176440    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176483    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176536    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176591    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176641    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.176806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.176736    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.231973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.231937    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" event={"ID":"bb32e360-3c70-43bd-a19a-86d25a2d0d3e","Type":"ContainerStarted","Data":"e8c66f3d10ad61f1f3ce807b2f0a2765479aea0f42ac4381dfc4802f63c36c7d"}
Oct 31 01:24:11.232119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.231996    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" event={"ID":"bb32e360-3c70-43bd-a19a-86d25a2d0d3e","Type":"ContainerStarted","Data":"da55b19e58e0aae10091706e1e0c26e76f8d38a8f9d5f49c4d9c10361cd297c8"}
Oct 31 01:24:11.277859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.277827    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.277872    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.277891    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwkkp\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.277911    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.277939    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278020    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278121    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278288    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278311    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278544    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278595    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.278709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.278673    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.279260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.279235    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.281085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.281062    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.281238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.281195    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.333916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.333840    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.334207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.334183    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwkkp\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.348903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.348845    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podStartSLOduration=1.348827881 podStartE2EDuration="1.348827881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:11.347400573 +0000 UTC m=+1172.648357167" watchObservedRunningTime="2026-10-31 01:24:11.348827881 +0000 UTC m=+1172.649784459"
Oct 31 01:24:11.362479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.362450    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:11.435810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.435776    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:11.436704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.436682    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:11.436800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.436732    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:11.511409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.510858    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"]
Oct 31 01:24:11.747759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.747734    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"]
Oct 31 01:24:11.748215 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:11.748191    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4c5e8b_a0aa_4df4_ba5d_fa42e690eef9.slice/crio-c745895d334a8333a50599a913cc3482a6fb6f625834eb7e8581c38125e15ed9 WatchSource:0}: Error finding container c745895d334a8333a50599a913cc3482a6fb6f625834eb7e8581c38125e15ed9: Status 404 returned error can't find the container with id c745895d334a8333a50599a913cc3482a6fb6f625834eb7e8581c38125e15ed9
Oct 31 01:24:11.750615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.750587    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:11.750682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.750645    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:11.750682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:11.750674    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:12.105279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.105225    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" podUID="5229b736-3aa1-4218-8817-ce7448d18447" containerName="payload-processing" containerID="cri-o://ba76bcf4eeac33e2d56614e86de09d1d0f2b97fc9f5afd67338a7a363210d8f7" gracePeriod=25
Oct 31 01:24:12.252866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.252831    5419 generic.go:356] "Generic (PLEG): container finished" podID="5229b736-3aa1-4218-8817-ce7448d18447" containerID="ba76bcf4eeac33e2d56614e86de09d1d0f2b97fc9f5afd67338a7a363210d8f7" exitCode=0
Oct 31 01:24:12.253297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.252903    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" event={"ID":"5229b736-3aa1-4218-8817-ce7448d18447","Type":"ContainerDied","Data":"ba76bcf4eeac33e2d56614e86de09d1d0f2b97fc9f5afd67338a7a363210d8f7"}
Oct 31 01:24:12.254508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.254483    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" event={"ID":"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9","Type":"ContainerStarted","Data":"0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5"}
Oct 31 01:24:12.254648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.254524    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" event={"ID":"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9","Type":"ContainerStarted","Data":"c745895d334a8333a50599a913cc3482a6fb6f625834eb7e8581c38125e15ed9"}
Oct 31 01:24:12.257255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.257198    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h" event={"ID":"91ac7064-20e8-4af6-90b0-a146b56e156c","Type":"ContainerStarted","Data":"0d1a4b007261f48e68d0a430908b63083891e6c256226b6355b0dae623f64666"}
Oct 31 01:24:12.257336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.257287    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h" event={"ID":"91ac7064-20e8-4af6-90b0-a146b56e156c","Type":"ContainerStarted","Data":"b959008a3640d8d511b35dbe82fbb99e581a06c0dae0d13a5205041eb493a16a"}
Oct 31 01:24:12.363609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.363531    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:12.364887 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.364860    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:12.365017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.364911    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:12.406367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.374507    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podStartSLOduration=1.374489896 podStartE2EDuration="1.374489896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:12.372741431 +0000 UTC m=+1173.673698009" watchObservedRunningTime="2026-10-31 01:24:12.374489896 +0000 UTC m=+1173.675446473"
Oct 31 01:24:12.436131 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.436089    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:12.436525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.436494    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:12.455846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.455822    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:24:12.521223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.514339    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h" podStartSLOduration=2.5143139899999998 podStartE2EDuration="2.51431399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:12.488777216 +0000 UTC m=+1173.789733800" watchObservedRunningTime="2026-10-31 01:24:12.51431399 +0000 UTC m=+1173.815270568"
Oct 31 01:24:12.622966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.622879    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume\") pod \"5229b736-3aa1-4218-8817-ce7448d18447\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") "
Oct 31 01:24:12.623248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.623228    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials\") pod \"5229b736-3aa1-4218-8817-ce7448d18447\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") "
Oct 31 01:24:12.623414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.623399    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume\") pod \"5229b736-3aa1-4218-8817-ce7448d18447\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") "
Oct 31 01:24:12.623523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.623510    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9\") pod \"5229b736-3aa1-4218-8817-ce7448d18447\" (UID: \"5229b736-3aa1-4218-8817-ce7448d18447\") "
Oct 31 01:24:12.628700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.627486    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume" pod "5229b736-3aa1-4218-8817-ce7448d18447" (UID: "5229b736-3aa1-4218-8817-ce7448d18447"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:12.628700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.627699    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume" pod "5229b736-3aa1-4218-8817-ce7448d18447" (UID: "5229b736-3aa1-4218-8817-ce7448d18447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:12.628700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.628640    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9" pod "5229b736-3aa1-4218-8817-ce7448d18447" (UID: "5229b736-3aa1-4218-8817-ce7448d18447"). InnerVolumeSpecName "kube-api-access-qsjj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:12.630923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.630895    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials" pod "5229b736-3aa1-4218-8817-ce7448d18447" (UID: "5229b736-3aa1-4218-8817-ce7448d18447"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:12.726181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.725661    5419 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-routing-overlay-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.726181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.725704    5419 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-provider-credentials\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.726181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.725724    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5229b736-3aa1-4218-8817-ce7448d18447-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.726181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:12.725738    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjj9\" (UniqueName: \"kubernetes.io/projected/5229b736-3aa1-4218-8817-ce7448d18447-kube-api-access-qsjj9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:13.302745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.298852    5419 generic.go:356] "Generic (PLEG): container finished" podID="91ac7064-20e8-4af6-90b0-a146b56e156c" containerID="0d1a4b007261f48e68d0a430908b63083891e6c256226b6355b0dae623f64666" exitCode=0
Oct 31 01:24:13.302745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.299196    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h" event={"ID":"91ac7064-20e8-4af6-90b0-a146b56e156c","Type":"ContainerDied","Data":"0d1a4b007261f48e68d0a430908b63083891e6c256226b6355b0dae623f64666"}
Oct 31 01:24:13.302745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.302461    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt" event={"ID":"5229b736-3aa1-4218-8817-ce7448d18447","Type":"ContainerDied","Data":"bce699b4b63e10d75d898031b3e1f9b1e61bce9e7522923907dda58011362bf1"}
Oct 31 01:24:13.302745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.302596    5419 scope.go:122] "RemoveContainer" containerID="ba76bcf4eeac33e2d56614e86de09d1d0f2b97fc9f5afd67338a7a363210d8f7"
Oct 31 01:24:13.303421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.302821    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"
Oct 31 01:24:13.364793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.364378    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:13.364793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.364435    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:13.435258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.435221    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:13.435527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.435497    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:13.541993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.541963    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"]
Oct 31 01:24:13.547359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:13.547338    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5dcfb47976-czmqt"]
Oct 31 01:24:14.363887 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.363853    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:14.364315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.363905    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:14.435678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.435635    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:14.435863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.435696    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:14.590405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.590328    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:14.633535 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:24:14.633490    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:24:14.754367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.754342    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p\" (UniqueName: \"kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p\") pod \"91ac7064-20e8-4af6-90b0-a146b56e156c\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") "
Oct 31 01:24:14.754846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.754830    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca\") pod \"91ac7064-20e8-4af6-90b0-a146b56e156c\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") "
Oct 31 01:24:14.755236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.755223    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca\") pod \"91ac7064-20e8-4af6-90b0-a146b56e156c\" (UID: \"91ac7064-20e8-4af6-90b0-a146b56e156c\") "
Oct 31 01:24:14.755651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.755623    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca" pod "91ac7064-20e8-4af6-90b0-a146b56e156c" (UID: "91ac7064-20e8-4af6-90b0-a146b56e156c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:14.755865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.755852    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:14.757298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.757283    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p" pod "91ac7064-20e8-4af6-90b0-a146b56e156c" (UID: "91ac7064-20e8-4af6-90b0-a146b56e156c"). InnerVolumeSpecName "kube-api-access-tzp8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:14.757388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.757287    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca" pod "91ac7064-20e8-4af6-90b0-a146b56e156c" (UID: "91ac7064-20e8-4af6-90b0-a146b56e156c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:14.837752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.837724    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5229b736-3aa1-4218-8817-ce7448d18447" path="/var/lib/kubelet/pods/5229b736-3aa1-4218-8817-ce7448d18447/volumes"
Oct 31 01:24:14.856682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.856646    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzp8p\" (UniqueName: \"kubernetes.io/projected/91ac7064-20e8-4af6-90b0-a146b56e156c-kube-api-access-tzp8p\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:14.856682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:14.856684    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ac7064-20e8-4af6-90b0-a146b56e156c-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:15.335168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.334733    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h" event={"ID":"91ac7064-20e8-4af6-90b0-a146b56e156c","Type":"ContainerDied","Data":"b959008a3640d8d511b35dbe82fbb99e581a06c0dae0d13a5205041eb493a16a"}
Oct 31 01:24:15.335168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.334821    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b959008a3640d8d511b35dbe82fbb99e581a06c0dae0d13a5205041eb493a16a"
Oct 31 01:24:15.335168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.334890    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"
Oct 31 01:24:15.363083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.363048    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:15.363236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.363097    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:15.436714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.436678    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:15.437123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:15.436742    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:16.362851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:16.362821    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:16.363063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:16.363044    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:16.435556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:16.435523    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:16.435741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:16.435579    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:17.363407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:17.363372    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:17.363813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:17.363439    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:17.435936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:17.435901    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 31 01:24:17.436113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:17.435961    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 31 01:24:18.366882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.366845    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 31 01:24:18.367428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.367394    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 31 01:24:18.439153 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.439103    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:18.439653 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.439634    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:18.440420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.440397    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:24:18.895344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.895305    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"]
Oct 31 01:24:18.896132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.896111    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5229b736-3aa1-4218-8817-ce7448d18447" containerName="payload-processing"
Oct 31 01:24:18.896217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.896156    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5229b736-3aa1-4218-8817-ce7448d18447" containerName="payload-processing"
Oct 31 01:24:18.896217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.896177    5419 cpu_manager.go:414] "containerMap: removing container" podUID="91ac7064-20e8-4af6-90b0-a146b56e156c" containerName="revoke-keys"
Oct 31 01:24:18.896217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.896186    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ac7064-20e8-4af6-90b0-a146b56e156c" containerName="revoke-keys"
Oct 31 01:24:18.896382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.896366    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5229b736-3aa1-4218-8817-ce7448d18447" containerName="payload-processing"
Oct 31 01:24:18.896442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.896386    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ac7064-20e8-4af6-90b0-a146b56e156c" containerName="revoke-keys"
Oct 31 01:24:18.901800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.901774    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:18.974388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:18.974359    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"]
Oct 31 01:24:19.048363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.047914    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048021    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048072    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048209    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048272    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048372    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048400    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fptm2\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048577    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.048770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.048676    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.149981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.149882    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.149993    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150078    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150110    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150161    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150223    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150292    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150333    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150362    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptm2\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150835    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.150918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.150876    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.151208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.151181    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.151392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.151366    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.151954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.151925    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.154174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.153824    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.154589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.154569    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.209344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.209304    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.209516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.209395    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptm2\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.220348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.220324    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:19.400166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.400069    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:19.400548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.400384    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:19.401057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.401039    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:24:19.688902 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:19.688868    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cebf8a_0c3f_470b_81fd_9c0952261b94.slice/crio-689c3af61fea9af9cc6c7ae0a4f9df36b46f38c39cd2b087a1f913c7714c56f2 WatchSource:0}: Error finding container 689c3af61fea9af9cc6c7ae0a4f9df36b46f38c39cd2b087a1f913c7714c56f2: Status 404 returned error can't find the container with id 689c3af61fea9af9cc6c7ae0a4f9df36b46f38c39cd2b087a1f913c7714c56f2
Oct 31 01:24:19.689914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.689877    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"]
Oct 31 01:24:19.692008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.691970    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:19.692117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.692052    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:19.692117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:19.692098    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:20.406406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:20.406370    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" event={"ID":"82cebf8a-0c3f-470b-81fd-9c0952261b94","Type":"ContainerStarted","Data":"68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085"}
Oct 31 01:24:20.406819 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:20.406427    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" event={"ID":"82cebf8a-0c3f-470b-81fd-9c0952261b94","Type":"ContainerStarted","Data":"689c3af61fea9af9cc6c7ae0a4f9df36b46f38c39cd2b087a1f913c7714c56f2"}
Oct 31 01:24:20.527607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:20.527553    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podStartSLOduration=2.527534075 podStartE2EDuration="2.527534075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:20.524958349 +0000 UTC m=+1181.825914926" watchObservedRunningTime="2026-10-31 01:24:20.527534075 +0000 UTC m=+1181.828490654"
Oct 31 01:24:21.238178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:21.237508    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:21.272180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:21.265282    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:21.272180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:21.265359    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:22.221604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.221568    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:22.222231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.222196    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:22.638327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.638223    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:22.780163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.779218    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:24:22.835292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.835239    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:22.926377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.926266    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:22.945891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.945109    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 01:24:22.980164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:22.975924    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:23.227364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.221673    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:23.227364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.221751    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:23.656850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.655340    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:23.656850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.656018    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:24:23.786342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.784220    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:24:23.786342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.784892    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:24:23.977225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.963122    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:23.977225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.963541    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:24:23.992164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.977620    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:23.992164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:23.978032    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:24:24.484778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.346753    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:24.484778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.346818    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:24.500389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.500351    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:24:24.500665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.500632    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.500890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.500870    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:24:24.501011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.500992    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.568949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.568899    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:24:24.571796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.571745    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.665006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.664957    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.666983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.666948    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.836220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.833574    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:24.997175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:24.995353    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:25.010391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.010349    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:25.231167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.227475    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:25.231167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.227545    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:25.308846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.308816    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:24:25.329126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.329101    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:24:25.341315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.341283    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:24:25.352910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:25.352889    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:24:26.220749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:26.220711    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:26.221263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:26.220773    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:27.221514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.221477    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 01:24:27.221901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.221530    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 01:24:27.643031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.642939    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"]
Oct 31 01:24:27.647384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.647358    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.704544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.704505    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"]
Oct 31 01:24:27.744414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.744371    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzbs\" (UniqueName: \"kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.744625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.744572    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.744716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.744696    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.748730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.748697    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:24:27.748866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.748766    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:24:27.748866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.748829    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:24:27.748984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.748907    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:24:27.846459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.846430    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.846665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.846515    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.846665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.846606    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzbs\" (UniqueName: \"kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.847208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.847176    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.849585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.849555    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.904132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.904050    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzbs\" (UniqueName: \"kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs\") pod \"maas-api-revoke-sub-24111e67-7n28q\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:27.960631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:27.960599    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:28.253302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.253270    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:28.253990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.253969    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:28.254951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.254929    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:24:28.359086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.359051    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"]
Oct 31 01:24:28.532880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.532695    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"]
Oct 31 01:24:28.538284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.538254    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.559482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.559448    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwbbb\" (UniqueName: \"kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.559625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.559554    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.559625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.559606    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.598004 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.597981    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"]
Oct 31 01:24:28.624928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.624878    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q" event={"ID":"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9","Type":"ContainerStarted","Data":"0d4600d2d6de5eb15be07a471d33285edf593c45113ed3c37242e8c7ecb27408"}
Oct 31 01:24:28.660398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.660367    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwbbb\" (UniqueName: \"kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.660603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.660412    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.660603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.660462    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.661073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.661050    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.662952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.662925    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.718038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.718006    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwbbb\" (UniqueName: \"kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb\") pod \"maas-api-revoke-sub-d3567e9c-4lrbr\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:28.855501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:28.855429    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:29.232049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:29.232025    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"]
Oct 31 01:24:29.233905 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:29.233877    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf691c212_8599_498e_9b1b_a867345b7cc0.slice/crio-cf71c5b373a0149829ff14c4f21a2f7d6d741720919fc77ae6715581ef2dcb0e WatchSource:0}: Error finding container cf71c5b373a0149829ff14c4f21a2f7d6d741720919fc77ae6715581ef2dcb0e: Status 404 returned error can't find the container with id cf71c5b373a0149829ff14c4f21a2f7d6d741720919fc77ae6715581ef2dcb0e
Oct 31 01:24:29.633681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:29.633644    5419 generic.go:356] "Generic (PLEG): container finished" podID="ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" containerID="68dd5f108222d348b7fe47d3cb9d0c02abbe3e6fe2a9a20522b72ef76e799412" exitCode=0
Oct 31 01:24:29.634128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:29.633718    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q" event={"ID":"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9","Type":"ContainerDied","Data":"68dd5f108222d348b7fe47d3cb9d0c02abbe3e6fe2a9a20522b72ef76e799412"}
Oct 31 01:24:29.635281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:29.635253    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr" event={"ID":"f691c212-8599-498e-9b1b-a867345b7cc0","Type":"ContainerStarted","Data":"cf71c5b373a0149829ff14c4f21a2f7d6d741720919fc77ae6715581ef2dcb0e"}
Oct 31 01:24:30.650028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.649922    5419 generic.go:356] "Generic (PLEG): container finished" podID="f691c212-8599-498e-9b1b-a867345b7cc0" containerID="2438425c89a245889ed764b4d234cbda05a66ef1442ba3cdf8236b3ffa5b900c" exitCode=0
Oct 31 01:24:30.650028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.649959    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr" event={"ID":"f691c212-8599-498e-9b1b-a867345b7cc0","Type":"ContainerDied","Data":"2438425c89a245889ed764b4d234cbda05a66ef1442ba3cdf8236b3ffa5b900c"}
Oct 31 01:24:30.815994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.815971    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:30.893616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.893591    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs\" (UniqueName: \"kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs\") pod \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") "
Oct 31 01:24:30.893771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.893748    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca\") pod \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") "
Oct 31 01:24:30.893813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.893792    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca\") pod \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\" (UID: \"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9\") "
Oct 31 01:24:30.894183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.894124    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca" pod "ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" (UID: "ccebd48e-7376-4c3a-9514-0d31fc7ba7c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:30.894304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.894282    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:30.895783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.895757    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs" pod "ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" (UID: "ccebd48e-7376-4c3a-9514-0d31fc7ba7c9"). InnerVolumeSpecName "kube-api-access-9dzbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:30.896065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.896046    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca" pod "ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" (UID: "ccebd48e-7376-4c3a-9514-0d31fc7ba7c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:30.996016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.995982    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:30.996016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:30.996011    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzbs\" (UniqueName: \"kubernetes.io/projected/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9-kube-api-access-9dzbs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:31.658980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.658940    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q" event={"ID":"ccebd48e-7376-4c3a-9514-0d31fc7ba7c9","Type":"ContainerDied","Data":"0d4600d2d6de5eb15be07a471d33285edf593c45113ed3c37242e8c7ecb27408"}
Oct 31 01:24:31.659483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.659009    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d4600d2d6de5eb15be07a471d33285edf593c45113ed3c37242e8c7ecb27408"
Oct 31 01:24:31.659483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.658956    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"
Oct 31 01:24:31.801895 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.801867    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:31.907563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.907534    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca\") pod \"f691c212-8599-498e-9b1b-a867345b7cc0\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") "
Oct 31 01:24:31.907710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.907695    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb\" (UniqueName: \"kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb\") pod \"f691c212-8599-498e-9b1b-a867345b7cc0\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") "
Oct 31 01:24:31.907775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.907741    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca\") pod \"f691c212-8599-498e-9b1b-a867345b7cc0\" (UID: \"f691c212-8599-498e-9b1b-a867345b7cc0\") "
Oct 31 01:24:31.908166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.908110    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca" pod "f691c212-8599-498e-9b1b-a867345b7cc0" (UID: "f691c212-8599-498e-9b1b-a867345b7cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:31.908296 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.908235    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:31.909750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.909689    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca" pod "f691c212-8599-498e-9b1b-a867345b7cc0" (UID: "f691c212-8599-498e-9b1b-a867345b7cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:31.909911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:31.909896    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb" pod "f691c212-8599-498e-9b1b-a867345b7cc0" (UID: "f691c212-8599-498e-9b1b-a867345b7cc0"). InnerVolumeSpecName "kube-api-access-pwbbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:32.009853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.009816    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwbbb\" (UniqueName: \"kubernetes.io/projected/f691c212-8599-498e-9b1b-a867345b7cc0-kube-api-access-pwbbb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:32.009853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.009853    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f691c212-8599-498e-9b1b-a867345b7cc0-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:32.047581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.047551    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"]
Oct 31 01:24:32.053013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.052985    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-019af69e-djl76"]
Oct 31 01:24:32.666127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.666098    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"
Oct 31 01:24:32.666531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.666125    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr" event={"ID":"f691c212-8599-498e-9b1b-a867345b7cc0","Type":"ContainerDied","Data":"cf71c5b373a0149829ff14c4f21a2f7d6d741720919fc77ae6715581ef2dcb0e"}
Oct 31 01:24:32.666531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.666221    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf71c5b373a0149829ff14c4f21a2f7d6d741720919fc77ae6715581ef2dcb0e"
Oct 31 01:24:32.837336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:32.837309    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e64e59-96bd-426c-a43b-ce25a2666532" path="/var/lib/kubelet/pods/c5e64e59-96bd-426c-a43b-ce25a2666532/volumes"
Oct 31 01:24:34.058793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:34.058760    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:24:35.321668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.321629    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"]
Oct 31 01:24:35.326522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.326494    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" containerName="revoke-keys"
Oct 31 01:24:35.326696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.326680    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" containerName="revoke-keys"
Oct 31 01:24:35.326865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.326853    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f691c212-8599-498e-9b1b-a867345b7cc0" containerName="revoke-keys"
Oct 31 01:24:35.326966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.326954    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f691c212-8599-498e-9b1b-a867345b7cc0" containerName="revoke-keys"
Oct 31 01:24:35.328746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.328725    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f691c212-8599-498e-9b1b-a867345b7cc0" containerName="revoke-keys"
Oct 31 01:24:35.328910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.328895    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" containerName="revoke-keys"
Oct 31 01:24:35.352249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.352218    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.369697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.369662    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"]
Oct 31 01:24:35.379412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.379377    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.381304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.381278    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgnx\" (UniqueName: \"kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.381568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.381549    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.406909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.406885    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:24:35.409059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.406806    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:24:35.409364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.409347    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:24:35.456423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.456392    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:24:35.483115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.482978    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.483953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.483831    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgnx\" (UniqueName: \"kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.484291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.484270    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.485586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.485235    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.502857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.502827    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.542947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.542915    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgnx\" (UniqueName: \"kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx\") pod \"maas-api-revoke-sub-8f027a4d-hj22g\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:35.705440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:35.705341    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:36.786975 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:36.786241    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:36.932803 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:36.932759    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:36.996082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:36.996049    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"]
Oct 31 01:24:37.883021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:37.882988    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g" event={"ID":"e3811269-2982-43fe-bde6-101768c44224","Type":"ContainerStarted","Data":"b8633c3092045224e7bce0e09e7305b4388d08ec2561dc31f8bd25f82272e48b"}
Oct 31 01:24:37.883439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:37.883051    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g" event={"ID":"e3811269-2982-43fe-bde6-101768c44224","Type":"ContainerStarted","Data":"dcecba392a2e53610152223221da6a9d19c7419b5b4592ab3e39bf5f825d7f6c"}
Oct 31 01:24:37.996993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:37.996941    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g" podStartSLOduration=2.996927501 podStartE2EDuration="2.996927501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:37.996086997 +0000 UTC m=+1199.297043576" watchObservedRunningTime="2026-10-31 01:24:37.996927501 +0000 UTC m=+1199.297884077"
Oct 31 01:24:38.413460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.413424    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"]
Oct 31 01:24:38.417418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.417401    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.468729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.468699    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.468903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.468762    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.468960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.468938    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnjk\" (UniqueName: \"kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.474921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.474895    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"]
Oct 31 01:24:38.570391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.570358    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnjk\" (UniqueName: \"kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.570545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.570459    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.570545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.570504    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.571243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.571218    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.572920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.572897    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.627794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.627765    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnjk\" (UniqueName: \"kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk\") pod \"maas-api-revoke-sub-f3a6708a-bmwq7\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.730403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.730375    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:38.890425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.890386    5419 generic.go:356] "Generic (PLEG): container finished" podID="e3811269-2982-43fe-bde6-101768c44224" containerID="b8633c3092045224e7bce0e09e7305b4388d08ec2561dc31f8bd25f82272e48b" exitCode=0
Oct 31 01:24:38.890804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:38.890468    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g" event={"ID":"e3811269-2982-43fe-bde6-101768c44224","Type":"ContainerDied","Data":"b8633c3092045224e7bce0e09e7305b4388d08ec2561dc31f8bd25f82272e48b"}
Oct 31 01:24:39.118086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.118055    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"]
Oct 31 01:24:39.118965 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:39.118936    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a6ed33_25a4_46bd_87d8_5e20797cd85a.slice/crio-3174207882f137f4f912e05af8acb8069aa8fa745b7189a38761b68b38e835c1 WatchSource:0}: Error finding container 3174207882f137f4f912e05af8acb8069aa8fa745b7189a38761b68b38e835c1: Status 404 returned error can't find the container with id 3174207882f137f4f912e05af8acb8069aa8fa745b7189a38761b68b38e835c1
Oct 31 01:24:39.124482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.124461    5419 scope.go:122] "RemoveContainer" containerID="038ab5a0181933b037a98fe52db622d37dadd0c637a3aac3c6eb1812e0f2bd1f"
Oct 31 01:24:39.321747 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.321720    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"]
Oct 31 01:24:39.327108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.327091    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.380130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.379259    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.380130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.379509    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvpn\" (UniqueName: \"kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.380130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.379587    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.382926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.382855    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"]
Oct 31 01:24:39.480372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.480336    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.480554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.480419    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.480554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.480538    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvpn\" (UniqueName: \"kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.481242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.481182    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.482954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.482917    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.570173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.557247    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvpn\" (UniqueName: \"kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn\") pod \"maas-api-revoke-sub-90ebd6f9-856kl\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.642026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.641948    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:39.898900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.898813    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7" event={"ID":"e0a6ed33-25a4-46bd-87d8-5e20797cd85a","Type":"ContainerStarted","Data":"0bce03a847a34c27887ad75d78a481861483a3b4e9c393422593063702e749a9"}
Oct 31 01:24:39.898900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:39.898874    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7" event={"ID":"e0a6ed33-25a4-46bd-87d8-5e20797cd85a","Type":"ContainerStarted","Data":"3174207882f137f4f912e05af8acb8069aa8fa745b7189a38761b68b38e835c1"}
Oct 31 01:24:40.012481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.012431    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7" podStartSLOduration=2.012414888 podStartE2EDuration="2.012414888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:40.011978104 +0000 UTC m=+1201.312934677" watchObservedRunningTime="2026-10-31 01:24:40.012414888 +0000 UTC m=+1201.313371464"
Oct 31 01:24:40.022177 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:40.022110    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a9bf4a_232e_484c_b26a_f5601f2e81a0.slice/crio-512c76df3455cf5ff422b876536470bc29d8f437dd89eb8650758b4d28eec6d1 WatchSource:0}: Error finding container 512c76df3455cf5ff422b876536470bc29d8f437dd89eb8650758b4d28eec6d1: Status 404 returned error can't find the container with id 512c76df3455cf5ff422b876536470bc29d8f437dd89eb8650758b4d28eec6d1
Oct 31 01:24:40.022784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.022757    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"]
Oct 31 01:24:40.031425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.028694    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:40.120768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.120743    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca\") pod \"e3811269-2982-43fe-bde6-101768c44224\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") "
Oct 31 01:24:40.120927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.120878    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca\") pod \"e3811269-2982-43fe-bde6-101768c44224\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") "
Oct 31 01:24:40.121006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.120944    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx\" (UniqueName: \"kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx\") pod \"e3811269-2982-43fe-bde6-101768c44224\" (UID: \"e3811269-2982-43fe-bde6-101768c44224\") "
Oct 31 01:24:40.121218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.121189    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca" pod "e3811269-2982-43fe-bde6-101768c44224" (UID: "e3811269-2982-43fe-bde6-101768c44224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:40.121361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.121343    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3811269-2982-43fe-bde6-101768c44224-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:40.123151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.123109    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca" pod "e3811269-2982-43fe-bde6-101768c44224" (UID: "e3811269-2982-43fe-bde6-101768c44224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:40.123250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.123184    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx" pod "e3811269-2982-43fe-bde6-101768c44224" (UID: "e3811269-2982-43fe-bde6-101768c44224"). InnerVolumeSpecName "kube-api-access-4qgnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:40.222425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.222394    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3811269-2982-43fe-bde6-101768c44224-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:40.222425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.222419    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qgnx\" (UniqueName: \"kubernetes.io/projected/e3811269-2982-43fe-bde6-101768c44224-kube-api-access-4qgnx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:40.906310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.906278    5419 generic.go:356] "Generic (PLEG): container finished" podID="e0a6ed33-25a4-46bd-87d8-5e20797cd85a" containerID="0bce03a847a34c27887ad75d78a481861483a3b4e9c393422593063702e749a9" exitCode=0
Oct 31 01:24:40.906693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.906365    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7" event={"ID":"e0a6ed33-25a4-46bd-87d8-5e20797cd85a","Type":"ContainerDied","Data":"0bce03a847a34c27887ad75d78a481861483a3b4e9c393422593063702e749a9"}
Oct 31 01:24:40.907686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.907668    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"
Oct 31 01:24:40.907799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.907702    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g" event={"ID":"e3811269-2982-43fe-bde6-101768c44224","Type":"ContainerDied","Data":"dcecba392a2e53610152223221da6a9d19c7419b5b4592ab3e39bf5f825d7f6c"}
Oct 31 01:24:40.907799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.907746    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcecba392a2e53610152223221da6a9d19c7419b5b4592ab3e39bf5f825d7f6c"
Oct 31 01:24:40.909308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.909287    5419 generic.go:356] "Generic (PLEG): container finished" podID="43a9bf4a-232e-484c-b26a-f5601f2e81a0" containerID="ee958fab662d298b4b5c6ffe8500aa4a95f22ac025319e1508b12a31d143397d" exitCode=0
Oct 31 01:24:40.909403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.909379    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl" event={"ID":"43a9bf4a-232e-484c-b26a-f5601f2e81a0","Type":"ContainerDied","Data":"ee958fab662d298b4b5c6ffe8500aa4a95f22ac025319e1508b12a31d143397d"}
Oct 31 01:24:40.909456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:40.909435    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl" event={"ID":"43a9bf4a-232e-484c-b26a-f5601f2e81a0","Type":"ContainerStarted","Data":"512c76df3455cf5ff422b876536470bc29d8f437dd89eb8650758b4d28eec6d1"}
Oct 31 01:24:41.073250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:41.073221    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"]
Oct 31 01:24:41.075201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:41.075178    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8c3125a-4sfc8"]
Oct 31 01:24:42.062930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.062906    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:42.106361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.106339    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:42.143941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.143916    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca\") pod \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") "
Oct 31 01:24:42.144072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.144004    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk\" (UniqueName: \"kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk\") pod \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") "
Oct 31 01:24:42.144072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.144053    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca\") pod \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\" (UID: \"e0a6ed33-25a4-46bd-87d8-5e20797cd85a\") "
Oct 31 01:24:42.144307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.144280    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca" pod "e0a6ed33-25a4-46bd-87d8-5e20797cd85a" (UID: "e0a6ed33-25a4-46bd-87d8-5e20797cd85a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:42.144597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.144578    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:42.146231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.146207    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca" pod "e0a6ed33-25a4-46bd-87d8-5e20797cd85a" (UID: "e0a6ed33-25a4-46bd-87d8-5e20797cd85a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:42.146332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.146264    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk" pod "e0a6ed33-25a4-46bd-87d8-5e20797cd85a" (UID: "e0a6ed33-25a4-46bd-87d8-5e20797cd85a"). InnerVolumeSpecName "kube-api-access-blnjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:42.245865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.245840    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca\") pod \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") "
Oct 31 01:24:42.246036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.245874    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca\") pod \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") "
Oct 31 01:24:42.246105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.246093    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn\" (UniqueName: \"kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn\") pod \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\" (UID: \"43a9bf4a-232e-484c-b26a-f5601f2e81a0\") "
Oct 31 01:24:42.246270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.246245    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca" pod "43a9bf4a-232e-484c-b26a-f5601f2e81a0" (UID: "43a9bf4a-232e-484c-b26a-f5601f2e81a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:42.246551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.246535    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:42.246597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.246555    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnjk\" (UniqueName: \"kubernetes.io/projected/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-kube-api-access-blnjk\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:42.246597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.246571    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a6ed33-25a4-46bd-87d8-5e20797cd85a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:42.248100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.248079    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca" pod "43a9bf4a-232e-484c-b26a-f5601f2e81a0" (UID: "43a9bf4a-232e-484c-b26a-f5601f2e81a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:42.248227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.248207    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn" pod "43a9bf4a-232e-484c-b26a-f5601f2e81a0" (UID: "43a9bf4a-232e-484c-b26a-f5601f2e81a0"). InnerVolumeSpecName "kube-api-access-wcvpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:42.348018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.347981    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvpn\" (UniqueName: \"kubernetes.io/projected/43a9bf4a-232e-484c-b26a-f5601f2e81a0-kube-api-access-wcvpn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:42.348018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.348010    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a9bf4a-232e-484c-b26a-f5601f2e81a0-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:24:42.837453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.837422    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90219e1d-896a-45c8-adca-9b54caf26764" path="/var/lib/kubelet/pods/90219e1d-896a-45c8-adca-9b54caf26764/volumes"
Oct 31 01:24:42.928005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.924725    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"
Oct 31 01:24:42.928005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.924761    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl" event={"ID":"43a9bf4a-232e-484c-b26a-f5601f2e81a0","Type":"ContainerDied","Data":"512c76df3455cf5ff422b876536470bc29d8f437dd89eb8650758b4d28eec6d1"}
Oct 31 01:24:42.928005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.924818    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512c76df3455cf5ff422b876536470bc29d8f437dd89eb8650758b4d28eec6d1"
Oct 31 01:24:42.928005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.926410    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7" event={"ID":"e0a6ed33-25a4-46bd-87d8-5e20797cd85a","Type":"ContainerDied","Data":"3174207882f137f4f912e05af8acb8069aa8fa745b7189a38761b68b38e835c1"}
Oct 31 01:24:42.928005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.926427    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"
Oct 31 01:24:42.928005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:42.926450    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3174207882f137f4f912e05af8acb8069aa8fa745b7189a38761b68b38e835c1"
Oct 31 01:24:44.581950 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:24:44.581909    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:24:46.391746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.391708    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"]
Oct 31 01:24:46.392294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392279    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e3811269-2982-43fe-bde6-101768c44224" containerName="revoke-keys"
Oct 31 01:24:46.392357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392299    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3811269-2982-43fe-bde6-101768c44224" containerName="revoke-keys"
Oct 31 01:24:46.392357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392317    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e0a6ed33-25a4-46bd-87d8-5e20797cd85a" containerName="revoke-keys"
Oct 31 01:24:46.392357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392324    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a6ed33-25a4-46bd-87d8-5e20797cd85a" containerName="revoke-keys"
Oct 31 01:24:46.392357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392344    5419 cpu_manager.go:414] "containerMap: removing container" podUID="43a9bf4a-232e-484c-b26a-f5601f2e81a0" containerName="revoke-keys"
Oct 31 01:24:46.392357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392350    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a9bf4a-232e-484c-b26a-f5601f2e81a0" containerName="revoke-keys"
Oct 31 01:24:46.392495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392427    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3811269-2982-43fe-bde6-101768c44224" containerName="revoke-keys"
Oct 31 01:24:46.392495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392434    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a6ed33-25a4-46bd-87d8-5e20797cd85a" containerName="revoke-keys"
Oct 31 01:24:46.392495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.392444    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a9bf4a-232e-484c-b26a-f5601f2e81a0" containerName="revoke-keys"
Oct 31 01:24:46.397232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.397217    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.451809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.451776    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"]
Oct 31 01:24:46.497093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.497061    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-00e840\"/\"distinct-w2-00e840-kserve-self-signed-certs\""
Oct 31 01:24:46.506854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.506829    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.506961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.506908    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.507018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.506996    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.507050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.507032    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.507102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.507055    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxtc\" (UniqueName: \"kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.507219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.507201    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.629068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.629025    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.629428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.629400    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.629650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.629631    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.629776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.629761    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.629882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.629866    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxtc\" (UniqueName: \"kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.630005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.629991    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.630615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.630589    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.630909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.630888    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.632850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.632826    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.643382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.643308    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.651458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.651411    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.689688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.689656    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxtc\" (UniqueName: \"kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc\") pod \"distinct-w2-00e840-kserve-7865645bcd-7vlp4\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") " pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:46.709917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:46.709895    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:47.087491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:47.087460    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"]
Oct 31 01:24:47.088094 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:47.088070    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cd2af97_bc5d_4af8_abf9_d6363d04233d.slice/crio-0204193a4a3572e4d18875c09f14418eb87e74d7601ccb4e8037ca13d3cefaeb WatchSource:0}: Error finding container 0204193a4a3572e4d18875c09f14418eb87e74d7601ccb4e8037ca13d3cefaeb: Status 404 returned error can't find the container with id 0204193a4a3572e4d18875c09f14418eb87e74d7601ccb4e8037ca13d3cefaeb
Oct 31 01:24:48.078169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.076653    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"]
Oct 31 01:24:48.094176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.093674    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" event={"ID":"7cd2af97-bc5d-4af8-abf9-d6363d04233d","Type":"ContainerStarted","Data":"132da53ac792fff1ba7922ec108fc88811610e6c06d0d189bae8310e193729a8"}
Oct 31 01:24:48.094176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.093731    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" event={"ID":"7cd2af97-bc5d-4af8-abf9-d6363d04233d","Type":"ContainerStarted","Data":"0204193a4a3572e4d18875c09f14418eb87e74d7601ccb4e8037ca13d3cefaeb"}
Oct 31 01:24:48.116162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.112827    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5089baf5-g2g67"]
Oct 31 01:24:48.350217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.350082    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podStartSLOduration=2.350061343 podStartE2EDuration="2.350061343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:48.230038754 +0000 UTC m=+1209.530995332" watchObservedRunningTime="2026-10-31 01:24:48.350061343 +0000 UTC m=+1209.651017922"
Oct 31 01:24:48.367110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.367058    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"]
Oct 31 01:24:48.382246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.380572    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.574627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.574593    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-18bf762c-openshift-default-dockercfg-8g8p2\""
Oct 31 01:24:48.645622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.638932    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"]
Oct 31 01:24:48.781661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.781633    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:48.781925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.697869    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.790526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.790496    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.790796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.790759    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.791004 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.790988    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.809164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.791456    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.809829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.809805    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.810093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.810074    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.810330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.810308    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7d9\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.810614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.810597    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.914894    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.914944    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.914973    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7d9\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915040    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915105    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915164    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915205    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915274    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915370    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.916216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.915981    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.917417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.917062    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.939183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.919235    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.939183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.919493    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:48.939183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.919743    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:49.013166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:48.987709    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:49.016622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:49.016589    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:49.021496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:49.021431    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:49.042166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:49.039661    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4342de72-0f8c-4eea-b454-bd2aa9b82fb3" path="/var/lib/kubelet/pods/4342de72-0f8c-4eea-b454-bd2aa9b82fb3/volumes"
Oct 31 01:24:49.042166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:49.040762    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7d9\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9\") pod \"e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") " pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:49.234274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:49.234194    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:49.341085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:49.341036    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:50.321405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:50.321358    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:50.562440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:50.562395    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:24:50.562643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:50.562456    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:50.656164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:50.655927    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:50.681315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:50.681269    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:50.939213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:50.923626    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:51.043937 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:51.043883    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode911c5ac_ba30_4372_8da0_7290fd15838c.slice/crio-a6ceb44184ceb396eb185deaf416f70bc5a356ad385ac29931a93a54f385b2dd WatchSource:0}: Error finding container a6ceb44184ceb396eb185deaf416f70bc5a356ad385ac29931a93a54f385b2dd: Status 404 returned error can't find the container with id a6ceb44184ceb396eb185deaf416f70bc5a356ad385ac29931a93a54f385b2dd
Oct 31 01:24:51.056968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.044462    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:24:51.065973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.065932    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:51.066359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.066330    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:51.066534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.066516    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:51.155180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.150362    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"]
Oct 31 01:24:51.652339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.652250    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"]
Oct 31 01:24:51.662204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.662173    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.708401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.708367    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" event={"ID":"e911c5ac-ba30-4372-8da0-7290fd15838c","Type":"ContainerStarted","Data":"7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"}
Oct 31 01:24:51.708578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.708433    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" event={"ID":"e911c5ac-ba30-4372-8da0-7290fd15838c","Type":"ContainerStarted","Data":"a6ceb44184ceb396eb185deaf416f70bc5a356ad385ac29931a93a54f385b2dd"}
Oct 31 01:24:51.730961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.730927    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzrf\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.730966    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731040    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731063    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731110    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731338    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731473    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731511    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.731551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.731540    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832331    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832434    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832466    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832596    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832677    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzrf\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832713    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832769    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832801    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832888    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.832968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.832902    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.833066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.833000    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.833263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.833232    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.833263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.833239    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.833564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.833540    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.835349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.835326    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.835580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.835563    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.867787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.867760    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"]
Oct 31 01:24:51.891818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.891788    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.891941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.891904    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzrf\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:51.897486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.897441    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podStartSLOduration=3.897428929 podStartE2EDuration="3.897428929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:51.894693546 +0000 UTC m=+1213.195650149" watchObservedRunningTime="2026-10-31 01:24:51.897428929 +0000 UTC m=+1213.198385584"
Oct 31 01:24:51.981101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:51.981074    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:52.235209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.235083    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:52.240608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.240579    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:52.520884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.520857    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"]
Oct 31 01:24:52.521775 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:24:52.521750    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92093e7_11b5_4988_ab07_ed138ce067f1.slice/crio-6daca443959f65a89a94c103dc1c3a8b60d1b43f535bef465cc7e5eb17ac6081 WatchSource:0}: Error finding container 6daca443959f65a89a94c103dc1c3a8b60d1b43f535bef465cc7e5eb17ac6081: Status 404 returned error can't find the container with id 6daca443959f65a89a94c103dc1c3a8b60d1b43f535bef465cc7e5eb17ac6081
Oct 31 01:24:52.524380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.524350    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:52.524456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.524416    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:52.524492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.524453    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:24:52.717050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.717013    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" event={"ID":"e92093e7-11b5-4988-ab07-ed138ce067f1","Type":"ContainerStarted","Data":"5aa53c8e1427b19c19c0255fff58f7046590931c8be268b9c053510b809b1381"}
Oct 31 01:24:52.717532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.717075    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" event={"ID":"e92093e7-11b5-4988-ab07-ed138ce067f1","Type":"ContainerStarted","Data":"6daca443959f65a89a94c103dc1c3a8b60d1b43f535bef465cc7e5eb17ac6081"}
Oct 31 01:24:52.717532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.717489    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:52.718058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.718040    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:24:52.836545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.836446    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podStartSLOduration=1.836433177 podStartE2EDuration="1.836433177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:52.832108822 +0000 UTC m=+1214.133065400" watchObservedRunningTime="2026-10-31 01:24:52.836433177 +0000 UTC m=+1214.137389813"
Oct 31 01:24:52.981997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:52.981963    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:53.045781 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:53.045752    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"]
Oct 31 01:24:53.048859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:53.048834    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7bec8f7-wplwt"]
Oct 31 01:24:54.000643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:54.000606    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:24:54.001052 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:54.000674    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:24:54.839754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:54.839711    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a" path="/var/lib/kubelet/pods/b4cf3e7f-b2d2-4ef7-b4f2-2640ad9fce6a/volumes"
Oct 31 01:24:54.999640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:54.995545    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:24:54.999640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:54.995609    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:24:55.983510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:55.983473    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:24:55.984118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:55.984087    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:24:56.302005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.301923    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"]
Oct 31 01:24:56.306014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.305995    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.362199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.362169    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"]
Oct 31 01:24:56.406202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.406168    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:24:56.406202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.406177    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:24:56.408474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.408453    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:24:56.408474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.408455    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:24:56.419151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.419117    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfxr\" (UniqueName: \"kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.419379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.419353    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.420036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.420012    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.521808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.521774    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfxr\" (UniqueName: \"kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.522005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.521948    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.522005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.522000    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.522657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.522634    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.524496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.524475    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.598207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.597808    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfxr\" (UniqueName: \"kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr\") pod \"maas-api-revoke-sub-b8686b2b-lngbk\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.619760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.619726    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:56.712517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.712487    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:56.729658 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.729431    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:56.759182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.759112    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:24:56.992524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.992102    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:24:56.992524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:56.992178    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:24:57.026595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:57.026563    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"]
Oct 31 01:24:57.756704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:57.756663    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk" event={"ID":"257c8c51-806f-4fca-a2ff-e68d0c064349","Type":"ContainerStarted","Data":"b2c27dced0b558e7a59590589b44b0b9d6706c6577a8b952f45eab23a53063a2"}
Oct 31 01:24:57.756889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:57.756729    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk" event={"ID":"257c8c51-806f-4fca-a2ff-e68d0c064349","Type":"ContainerStarted","Data":"e0dd70598dd99d10cabaaf65d43b74ccfb369656d091354f8d0cfa906a2cd875"}
Oct 31 01:24:57.872527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:57.872465    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk" podStartSLOduration=1.87245063 podStartE2EDuration="1.87245063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:57.870238504 +0000 UTC m=+1219.171195083" watchObservedRunningTime="2026-10-31 01:24:57.87245063 +0000 UTC m=+1219.173407206"
Oct 31 01:24:57.991762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:57.991727    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:24:57.992006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:57.991978    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:24:58.767802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:58.767765    5419 generic.go:356] "Generic (PLEG): container finished" podID="257c8c51-806f-4fca-a2ff-e68d0c064349" containerID="b2c27dced0b558e7a59590589b44b0b9d6706c6577a8b952f45eab23a53063a2" exitCode=0
Oct 31 01:24:58.768246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:58.767853    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk" event={"ID":"257c8c51-806f-4fca-a2ff-e68d0c064349","Type":"ContainerDied","Data":"b2c27dced0b558e7a59590589b44b0b9d6706c6577a8b952f45eab23a53063a2"}
Oct 31 01:24:58.981659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:58.981625    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 31 01:24:58.981798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:58.981691    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 31 01:24:59.930607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.930579    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:24:59.966414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.966383    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca\") pod \"257c8c51-806f-4fca-a2ff-e68d0c064349\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") "
Oct 31 01:24:59.966554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.966444    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr\" (UniqueName: \"kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr\") pod \"257c8c51-806f-4fca-a2ff-e68d0c064349\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") "
Oct 31 01:24:59.966554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.966506    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca\") pod \"257c8c51-806f-4fca-a2ff-e68d0c064349\" (UID: \"257c8c51-806f-4fca-a2ff-e68d0c064349\") "
Oct 31 01:24:59.966878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.966854    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca" pod "257c8c51-806f-4fca-a2ff-e68d0c064349" (UID: "257c8c51-806f-4fca-a2ff-e68d0c064349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:59.968538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.968512    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca" pod "257c8c51-806f-4fca-a2ff-e68d0c064349" (UID: "257c8c51-806f-4fca-a2ff-e68d0c064349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:24:59.968633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.968624    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr" pod "257c8c51-806f-4fca-a2ff-e68d0c064349" (UID: "257c8c51-806f-4fca-a2ff-e68d0c064349"). InnerVolumeSpecName "kube-api-access-8zfxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:59.986637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.986615    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:59.986908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.986882    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:24:59.987609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:24:59.987593    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:25:00.067631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:00.067541    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:00.067631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:00.067574    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfxr\" (UniqueName: \"kubernetes.io/projected/257c8c51-806f-4fca-a2ff-e68d0c064349-kube-api-access-8zfxr\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:00.067631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:00.067626    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c8c51-806f-4fca-a2ff-e68d0c064349-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:00.790731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:00.790701    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"
Oct 31 01:25:00.790889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:00.790742    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk" event={"ID":"257c8c51-806f-4fca-a2ff-e68d0c064349","Type":"ContainerDied","Data":"e0dd70598dd99d10cabaaf65d43b74ccfb369656d091354f8d0cfa906a2cd875"}
Oct 31 01:25:00.790889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:00.790816    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0dd70598dd99d10cabaaf65d43b74ccfb369656d091354f8d0cfa906a2cd875"
Oct 31 01:25:01.855503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:01.855470    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:25:04.269799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.269761    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"]
Oct 31 01:25:04.271276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.271250    5419 cpu_manager.go:414] "containerMap: removing container" podUID="257c8c51-806f-4fca-a2ff-e68d0c064349" containerName="revoke-keys"
Oct 31 01:25:04.280231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.279230    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257c8c51-806f-4fca-a2ff-e68d0c064349" containerName="revoke-keys"
Oct 31 01:25:04.280231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.279461    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="257c8c51-806f-4fca-a2ff-e68d0c064349" containerName="revoke-keys"
Oct 31 01:25:04.290211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.290182    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.316507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.311243    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321216    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321296    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321330    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321371    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321433    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321458    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s8p\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321536    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321577    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.329181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.321642    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.333255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.333216    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.352168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.351981    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.404160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.402573    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"]
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.426841    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.426894    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s8p\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.426979    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427021    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427084    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427198    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427261    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427293    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427337    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.437173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.427883    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.475166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.463505    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.475166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.464186    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.475166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.464489    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.475166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.464736    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.531212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.517701    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.605170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.566561    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.605170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.593439    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.631403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.630901    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s8p\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.631403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.631176    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:04.674168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.648930    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.674998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.674576    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.754050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.753233    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:04.952172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:04.951746    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:25:05.334711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.334572    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:05.338198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.338174    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:25:05.363300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.363258    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:05.364025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.363902    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:25:05.365055 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.365007    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:05.365499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.365479    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:25:05.422163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.413595    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:25:05.521159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.519112    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:25:05.556464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.555863    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 01:25:05.665688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.665651    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"]
Oct 31 01:25:05.666655 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:25:05.666633    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b40557c_6f00_44c2_9332_f94ba4998905.slice/crio-76f17466c8bc5efa8c9d4382d427cccd0ce1d8e5a5b46b4f26fad1c55dcb4bac WatchSource:0}: Error finding container 76f17466c8bc5efa8c9d4382d427cccd0ce1d8e5a5b46b4f26fad1c55dcb4bac: Status 404 returned error can't find the container with id 76f17466c8bc5efa8c9d4382d427cccd0ce1d8e5a5b46b4f26fad1c55dcb4bac
Oct 31 01:25:05.669809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.669773    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:25:05.669892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.669856    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:25:05.669945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:05.669901    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:25:06.009256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:06.009221    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" event={"ID":"1b40557c-6f00-44c2-9332-f94ba4998905","Type":"ContainerStarted","Data":"d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb"}
Oct 31 01:25:06.009434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:06.009294    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" event={"ID":"1b40557c-6f00-44c2-9332-f94ba4998905","Type":"ContainerStarted","Data":"76f17466c8bc5efa8c9d4382d427cccd0ce1d8e5a5b46b4f26fad1c55dcb4bac"}
Oct 31 01:25:06.632451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:06.632412    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:06.634172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:06.634127    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:06.634300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:06.634187    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:07.102699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.102637    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podStartSLOduration=3.102618061 podStartE2EDuration="3.102618061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:06.136890618 +0000 UTC m=+1227.437847196" watchObservedRunningTime="2026-10-31 01:25:07.102618061 +0000 UTC m=+1228.403574643"
Oct 31 01:25:07.105260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.105235    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"]
Oct 31 01:25:07.109424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.109407    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.157317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.157286    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zcvg\" (UniqueName: \"kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.157457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.157346    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.157655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.157633    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.158151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.158112    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"]
Oct 31 01:25:07.203264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.203229    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:25:07.203506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.203394    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:25:07.203506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.203421    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:25:07.210319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.210301    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:25:07.258634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.258602    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.258755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.258648    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zcvg\" (UniqueName: \"kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.258755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.258705    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.259472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.259450    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.261272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.261256    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.314891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.314861    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zcvg\" (UniqueName: \"kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg\") pod \"maas-api-revoke-sub-f597032c-4wzgf\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.423686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.423598    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:07.633059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.633028    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:07.633455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.633082    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:07.817913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:07.817872    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"]
Oct 31 01:25:07.818456 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:25:07.818430    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632e3055_6be7_49dd_9edb_b16e13d17d72.slice/crio-d0d9bc662cf4fbc0bc96c38f056386b9c6c2c6341e9c5dab150d639fa89ebe7e WatchSource:0}: Error finding container d0d9bc662cf4fbc0bc96c38f056386b9c6c2c6341e9c5dab150d639fa89ebe7e: Status 404 returned error can't find the container with id d0d9bc662cf4fbc0bc96c38f056386b9c6c2c6341e9c5dab150d639fa89ebe7e
Oct 31 01:25:08.025206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:08.025167    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf" event={"ID":"632e3055-6be7-49dd-9edb-b16e13d17d72","Type":"ContainerStarted","Data":"d0d9bc662cf4fbc0bc96c38f056386b9c6c2c6341e9c5dab150d639fa89ebe7e"}
Oct 31 01:25:08.637652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:08.637611    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:08.638929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:08.638894    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:09.234358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:09.234328    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"]
Oct 31 01:25:09.238961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:09.238937    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1a85e88-pfsq6"]
Oct 31 01:25:09.241220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:09.241185    5419 generic.go:356] "Generic (PLEG): container finished" podID="632e3055-6be7-49dd-9edb-b16e13d17d72" containerID="a6a674a84e9ba96ca142490918a6f2f03e3fcad8a7a46876425cbaf3435ac40f" exitCode=0
Oct 31 01:25:09.241376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:09.241280    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf" event={"ID":"632e3055-6be7-49dd-9edb-b16e13d17d72","Type":"ContainerDied","Data":"a6a674a84e9ba96ca142490918a6f2f03e3fcad8a7a46876425cbaf3435ac40f"}
Oct 31 01:25:09.632886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:09.632804    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:09.632886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:09.632857    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:10.499680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.499646    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:10.632820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.632740    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:10.632977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.632807    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:10.646604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.646576    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg\" (UniqueName: \"kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg\") pod \"632e3055-6be7-49dd-9edb-b16e13d17d72\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") "
Oct 31 01:25:10.646765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.646648    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca\") pod \"632e3055-6be7-49dd-9edb-b16e13d17d72\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") "
Oct 31 01:25:10.646828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.646789    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca\") pod \"632e3055-6be7-49dd-9edb-b16e13d17d72\" (UID: \"632e3055-6be7-49dd-9edb-b16e13d17d72\") "
Oct 31 01:25:10.647029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.646997    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca" pod "632e3055-6be7-49dd-9edb-b16e13d17d72" (UID: "632e3055-6be7-49dd-9edb-b16e13d17d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:25:10.647346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.647328    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:10.649475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.649447    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca" pod "632e3055-6be7-49dd-9edb-b16e13d17d72" (UID: "632e3055-6be7-49dd-9edb-b16e13d17d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:10.649574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.649553    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg" pod "632e3055-6be7-49dd-9edb-b16e13d17d72" (UID: "632e3055-6be7-49dd-9edb-b16e13d17d72"). InnerVolumeSpecName "kube-api-access-7zcvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:10.748821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.748787    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zcvg\" (UniqueName: \"kubernetes.io/projected/632e3055-6be7-49dd-9edb-b16e13d17d72-kube-api-access-7zcvg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:10.748821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.748824    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/632e3055-6be7-49dd-9edb-b16e13d17d72-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:10.838057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:10.838022    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9daaaaf-047e-40b9-ab47-b01ad52e7e1f" path="/var/lib/kubelet/pods/d9daaaaf-047e-40b9-ab47-b01ad52e7e1f/volumes"
Oct 31 01:25:11.236528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:11.236492    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"]
Oct 31 01:25:11.411313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:11.398399    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf" event={"ID":"632e3055-6be7-49dd-9edb-b16e13d17d72","Type":"ContainerDied","Data":"d0d9bc662cf4fbc0bc96c38f056386b9c6c2c6341e9c5dab150d639fa89ebe7e"}
Oct 31 01:25:11.411633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:11.411604    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d9bc662cf4fbc0bc96c38f056386b9c6c2c6341e9c5dab150d639fa89ebe7e"
Oct 31 01:25:11.411793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:11.400281    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"
Oct 31 01:25:11.632596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:11.632511    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:11.632596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:11.632584    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:12.632770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:12.632734    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 31 01:25:12.633297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:12.632797    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 31 01:25:13.636864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:13.636837    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:13.637293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:13.637236    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:13.638006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:13.637988    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:25:14.582847 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:25:14.582814    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:25:19.392516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.392484    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"]
Oct 31 01:25:19.393124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.393105    5419 cpu_manager.go:414] "containerMap: removing container" podUID="632e3055-6be7-49dd-9edb-b16e13d17d72" containerName="revoke-keys"
Oct 31 01:25:19.393214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.393127    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632e3055-6be7-49dd-9edb-b16e13d17d72" containerName="revoke-keys"
Oct 31 01:25:19.393292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.393282    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="632e3055-6be7-49dd-9edb-b16e13d17d72" containerName="revoke-keys"
Oct 31 01:25:19.398337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.398317    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.548634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548594    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.548634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548637    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.548898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548661    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.548898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548764    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmwtz\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.548898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548825    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.548898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548843    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.549124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548909    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.549124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548934    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.549124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.548968    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.596388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.596351    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"]
Oct 31 01:25:19.650275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650199    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650243    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650282    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650355    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650376    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650400    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650457    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmwtz\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650547    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650572    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650609    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650666    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.650939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650826    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.651008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.650988    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.651290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.651273    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.653492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.653467    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.653605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.653579    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.706633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.706599    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.706778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.706669    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmwtz\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz\") pod \"e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") " pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:19.712479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:19.712456    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:20.049168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.049122    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"]
Oct 31 01:25:20.052349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.052325    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b9e7cff-fgtt7"]
Oct 31 01:25:20.149732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.149700    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"]
Oct 31 01:25:20.150422 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:25:20.150397    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c77cdb9_63e8_4646_8c8a_1bdd8fcb2ff5.slice/crio-b1ee93dce81761591b14976bb3fe160e2fd60394fee036e14f44946b3836ef58 WatchSource:0}: Error finding container b1ee93dce81761591b14976bb3fe160e2fd60394fee036e14f44946b3836ef58: Status 404 returned error can't find the container with id b1ee93dce81761591b14976bb3fe160e2fd60394fee036e14f44946b3836ef58
Oct 31 01:25:20.153022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.152994    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:25:20.153090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.153057    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:25:20.153127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.153090    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:25:20.466961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.466861    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" event={"ID":"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5","Type":"ContainerStarted","Data":"2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c"}
Oct 31 01:25:20.466961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.466922    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" event={"ID":"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5","Type":"ContainerStarted","Data":"b1ee93dce81761591b14976bb3fe160e2fd60394fee036e14f44946b3836ef58"}
Oct 31 01:25:20.583821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.583767    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podStartSLOduration=1.58375117 podStartE2EDuration="1.58375117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:20.58142678 +0000 UTC m=+1241.882383371" watchObservedRunningTime="2026-10-31 01:25:20.58375117 +0000 UTC m=+1241.884707747"
Oct 31 01:25:20.713450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.713423    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:20.849446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:20.849411    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29da3141-21b8-4155-a281-f54be4e4d868" path="/var/lib/kubelet/pods/29da3141-21b8-4155-a281-f54be4e4d868/volumes"
Oct 31 01:25:21.714864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:21.714832    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:21.715418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:21.714879    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:22.714116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:22.714083    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:22.714465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:22.714422    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:23.726179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:23.721705    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:23.726179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:23.721792    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:24.420164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.419221    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 31 01:25:24.524161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.523068    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.561275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.561229    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.647277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.647223    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 31 01:25:24.674351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.674243    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.717794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.717756    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:24.718109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.718064    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:24.781165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.774287    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:24.928364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:24.928263    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:25.472388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:25.472343    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded"
Oct 31 01:25:25.473306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:25.473286    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:25:25.519418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:25.519388    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:25:25.713186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:25.713010    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:25.713186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:25.713055    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:26.427693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:26.427647    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main" containerID="cri-o://d7e178ed0e92cbe31406e9a4d888166655786f14b3b3470c887798c42dac9e37" gracePeriod=45
Oct 31 01:25:26.607428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:26.607178    5419 generic.go:356] "Generic (PLEG): container finished" podID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerID="d7e178ed0e92cbe31406e9a4d888166655786f14b3b3470c887798c42dac9e37" exitCode=0
Oct 31 01:25:26.607428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:26.607366    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" event={"ID":"c1ff6375-f75b-447b-b3f2-30184f1f94a2","Type":"ContainerDied","Data":"d7e178ed0e92cbe31406e9a4d888166655786f14b3b3470c887798c42dac9e37"}
Oct 31 01:25:26.715571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:26.715536    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:26.716764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:26.716370    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:26.830322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:26.830288    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:25:27.032127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032095    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir\") pod \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") "
Oct 31 01:25:27.032345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032163    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home\") pod \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") "
Oct 31 01:25:27.032345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032207    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs\") pod \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") "
Oct 31 01:25:27.032345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032284    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache\") pod \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") "
Oct 31 01:25:27.032525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032435    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm\") pod \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") "
Oct 31 01:25:27.032525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032483    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j\" (UniqueName: \"kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j\") pod \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\" (UID: \"c1ff6375-f75b-447b-b3f2-30184f1f94a2\") "
Oct 31 01:25:27.032744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032711    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache" pod "c1ff6375-f75b-447b-b3f2-30184f1f94a2" (UID: "c1ff6375-f75b-447b-b3f2-30184f1f94a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:27.032972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032788    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home" pod "c1ff6375-f75b-447b-b3f2-30184f1f94a2" (UID: "c1ff6375-f75b-447b-b3f2-30184f1f94a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:27.033095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.032976    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir" pod "c1ff6375-f75b-447b-b3f2-30184f1f94a2" (UID: "c1ff6375-f75b-447b-b3f2-30184f1f94a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:27.033203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.033184    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:27.035047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.035026    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm" pod "c1ff6375-f75b-447b-b3f2-30184f1f94a2" (UID: "c1ff6375-f75b-447b-b3f2-30184f1f94a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:25:27.035318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.035301    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs" pod "c1ff6375-f75b-447b-b3f2-30184f1f94a2" (UID: "c1ff6375-f75b-447b-b3f2-30184f1f94a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:27.035395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.035337    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j" pod "c1ff6375-f75b-447b-b3f2-30184f1f94a2" (UID: "c1ff6375-f75b-447b-b3f2-30184f1f94a2"). InnerVolumeSpecName "kube-api-access-fkc8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:27.134022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.133986    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:27.134022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.134018    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:27.134022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.134030    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ff6375-f75b-447b-b3f2-30184f1f94a2-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:27.134294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.134040    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1ff6375-f75b-447b-b3f2-30184f1f94a2-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:27.134294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.134053    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkc8j\" (UniqueName: \"kubernetes.io/projected/c1ff6375-f75b-447b-b3f2-30184f1f94a2-kube-api-access-fkc8j\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:27.616243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.616199    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"
Oct 31 01:25:27.616243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.616232    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd" event={"ID":"c1ff6375-f75b-447b-b3f2-30184f1f94a2","Type":"ContainerDied","Data":"cd4b8506dcb01264555eb5ed7ee71e2c45fa425ac65c192699b90f9c7593519f"}
Oct 31 01:25:27.616845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.616285    5419 scope.go:122] "RemoveContainer" containerID="d7e178ed0e92cbe31406e9a4d888166655786f14b3b3470c887798c42dac9e37"
Oct 31 01:25:27.713270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.713235    5419 patch_prober.go:29] interesting pod/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 01:25:27.713470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.713300    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 01:25:27.783371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.783338    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"]
Oct 31 01:25:27.786550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:27.786521    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/e2e-conflict-b-8bd02cd2-kserve-7c469796b4-gf9jd"]
Oct 31 01:25:28.717869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:28.717836    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:28.718301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:28.718249    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:28.718962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:28.718938    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:25:28.837875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:28.837843    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" path="/var/lib/kubelet/pods/c1ff6375-f75b-447b-b3f2-30184f1f94a2/volumes"
Oct 31 01:25:38.770073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.770039    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"]
Oct 31 01:25:38.770608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.770594    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main"
Oct 31 01:25:38.770898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.770609    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main"
Oct 31 01:25:38.770898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.770702    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ff6375-f75b-447b-b3f2-30184f1f94a2" containerName="main"
Oct 31 01:25:38.774212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.774196    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:38.875224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.875186    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:25:38.875426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.875228    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:25:38.875426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.875243    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:25:38.875426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.875319    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:25:38.879657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.879630    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"]
Oct 31 01:25:38.909437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.909408    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:38.909589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.909513    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:38.909786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:38.909765    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzzc\" (UniqueName: \"kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.014711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.014673    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.014909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.014868    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzzc\" (UniqueName: \"kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.015019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.015003    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.016039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.016010    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.024020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.023960    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.104163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.101948    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzzc\" (UniqueName: \"kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc\") pod \"maas-api-revoke-sub-f87266d5-pkfxj\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.168162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.167130    5419 scope.go:122] "RemoveContainer" containerID="4765449f254ac63977503c34a3300da4750e1e0b405266f8d3412facf090d785"
Oct 31 01:25:39.206500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.206465    5419 scope.go:122] "RemoveContainer" containerID="4d2d8a77c53fead96a69aca01b62a45869f62c650beff8e1975319bcd5af45f4"
Oct 31 01:25:39.217679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.217657    5419 scope.go:122] "RemoveContainer" containerID="ee044be81e42180181dffab872dbc6e6631a0b19dfe2cd6ea1a7ed691b741bf1"
Oct 31 01:25:39.387083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.387007    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:39.771900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:39.771872    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"]
Oct 31 01:25:39.772250 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:25:39.772201    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6c0141_26a9_473b_a1a2_7a26ba0ee952.slice/crio-afaba6dbc777c906615abe12a2446bf14ad54c932013c63ccb0fb920b4f452f0 WatchSource:0}: Error finding container afaba6dbc777c906615abe12a2446bf14ad54c932013c63ccb0fb920b4f452f0: Status 404 returned error can't find the container with id afaba6dbc777c906615abe12a2446bf14ad54c932013c63ccb0fb920b4f452f0
Oct 31 01:25:40.713244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:40.713212    5419 generic.go:356] "Generic (PLEG): container finished" podID="fd6c0141-26a9-473b-a1a2-7a26ba0ee952" containerID="2214573a464ff4ec8a311c56d72b8b7dc54b6f05721996cbe7eb853d3e194f99" exitCode=0
Oct 31 01:25:40.713420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:40.713258    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj" event={"ID":"fd6c0141-26a9-473b-a1a2-7a26ba0ee952","Type":"ContainerDied","Data":"2214573a464ff4ec8a311c56d72b8b7dc54b6f05721996cbe7eb853d3e194f99"}
Oct 31 01:25:40.713420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:40.713296    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj" event={"ID":"fd6c0141-26a9-473b-a1a2-7a26ba0ee952","Type":"ContainerStarted","Data":"afaba6dbc777c906615abe12a2446bf14ad54c932013c63ccb0fb920b4f452f0"}
Oct 31 01:25:41.860105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.860079    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:41.962494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.962454    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca\") pod \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") "
Oct 31 01:25:41.962494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.962495    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca\") pod \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") "
Oct 31 01:25:41.962749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.962656    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc\" (UniqueName: \"kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc\") pod \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\" (UID: \"fd6c0141-26a9-473b-a1a2-7a26ba0ee952\") "
Oct 31 01:25:41.962841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.962814    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca" pod "fd6c0141-26a9-473b-a1a2-7a26ba0ee952" (UID: "fd6c0141-26a9-473b-a1a2-7a26ba0ee952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:25:41.963308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.963286    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:41.965193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.965169    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc" pod "fd6c0141-26a9-473b-a1a2-7a26ba0ee952" (UID: "fd6c0141-26a9-473b-a1a2-7a26ba0ee952"). InnerVolumeSpecName "kube-api-access-6kzzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:25:41.965275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:41.965240    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca" pod "fd6c0141-26a9-473b-a1a2-7a26ba0ee952" (UID: "fd6c0141-26a9-473b-a1a2-7a26ba0ee952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:25:42.064551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:42.064511    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:42.064551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:42.064546    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzzc\" (UniqueName: \"kubernetes.io/projected/fd6c0141-26a9-473b-a1a2-7a26ba0ee952-kube-api-access-6kzzc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:25:42.727462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:42.727432    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"
Oct 31 01:25:42.727462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:42.727452    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj" event={"ID":"fd6c0141-26a9-473b-a1a2-7a26ba0ee952","Type":"ContainerDied","Data":"afaba6dbc777c906615abe12a2446bf14ad54c932013c63ccb0fb920b4f452f0"}
Oct 31 01:25:42.727642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:42.727499    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afaba6dbc777c906615abe12a2446bf14ad54c932013c63ccb0fb920b4f452f0"
Oct 31 01:25:44.581712 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:25:44.581673    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:25:53.459859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:53.459776    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:25:55.646167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:55.645293    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:55.670179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:55.669285    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:55.759167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:55.758332    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Oct 31 01:25:55.925299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:55.925203    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:25:56.662162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:56.660090    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:56.662162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:56.660554    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:25:56.672197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:56.670227    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:56.672197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:56.670671    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:25:56.764172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:56.763273    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:56.764172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:56.763715    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:25:57.096896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:57.096852    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc"
Oct 31 01:25:57.156797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:57.149215    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 01:25:57.440323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:57.440238    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 01:25:57.816161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:57.811516    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": context deadline exceeded"
Oct 31 01:25:58.215667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:58.202461    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:58.551167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:58.548545    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:25:58.570800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:25:58.570757    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:09.432164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:09.429303    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:09.432164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:09.429376    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:09.471163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:09.470252    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:09.471163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:09.470334    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:09.543162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:09.534307    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:09.543162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:09.534378    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:14.582680 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:26:14.582634    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:26:19.605088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:19.605044    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:26:23.758327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:23.758283    5419 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:23.758808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:23.758371    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:25.633185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:25.633154    5419 generic.go:356] "Generic (PLEG): container finished" podID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerID="7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44" exitCode=0
Oct 31 01:26:25.633185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:25.633163    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" event={"ID":"e911c5ac-ba30-4372-8da0-7290fd15838c","Type":"ContainerDied","Data":"7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"}
Oct 31 01:26:25.633641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:25.633542    5419 scope.go:122] "RemoveContainer" containerID="7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"
Oct 31 01:26:25.635096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:25.635071    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:26:25.635212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:25.635161    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:26:25.635212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:25.635196    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:26:26.648383 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:26.648329    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" event={"ID":"e911c5ac-ba30-4372-8da0-7290fd15838c","Type":"ContainerStarted","Data":"415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2"}
Oct 31 01:26:27.235231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.235195    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:26:27.235432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.235411    5419 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:26:27.235500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.235478    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 01:26:27.638866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.638788    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"]
Oct 31 01:26:27.639396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.639381    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fd6c0141-26a9-473b-a1a2-7a26ba0ee952" containerName="revoke-keys"
Oct 31 01:26:27.639446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.639397    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6c0141-26a9-473b-a1a2-7a26ba0ee952" containerName="revoke-keys"
Oct 31 01:26:27.639502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.639493    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6c0141-26a9-473b-a1a2-7a26ba0ee952" containerName="revoke-keys"
Oct 31 01:26:27.641661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.641647    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.740944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.740911    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.741455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.740960    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.741455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.741007    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.741455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.741051    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlzb\" (UniqueName: \"kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.741455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.741156    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.741455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.741304    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.749840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.749821    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8f6b5b\"/\"distinct-w4-8f6b5b-kserve-self-signed-certs\""
Oct 31 01:26:27.753779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.753760    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"]
Oct 31 01:26:27.848327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.848287    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.849204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849178    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849603    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849640    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlzb\" (UniqueName: \"kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849659    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849676    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849832    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.850331    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.850989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.849070    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.855174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.855052    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.855840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.855800    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.923285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.917528    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlzb\" (UniqueName: \"kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb\") pod \"distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") " pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:27.957357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:27.957325    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:28.248607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.248248    5419 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:26:28.248607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.248307    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 01:26:28.365090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.365066    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"]
Oct 31 01:26:28.366657 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:26:28.366627    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc80e2b1_5791_4e07_acde_514653cd7385.slice/crio-9887c12fc21666d17b6b1967c668cd55535655ae02d185fa01761b62abf43d6d WatchSource:0}: Error finding container 9887c12fc21666d17b6b1967c668cd55535655ae02d185fa01761b62abf43d6d: Status 404 returned error can't find the container with id 9887c12fc21666d17b6b1967c668cd55535655ae02d185fa01761b62abf43d6d
Oct 31 01:26:28.664810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.664726    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" event={"ID":"dc80e2b1-5791-4e07-acde-514653cd7385","Type":"ContainerStarted","Data":"1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0"}
Oct 31 01:26:28.664810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.664768    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" event={"ID":"dc80e2b1-5791-4e07-acde-514653cd7385","Type":"ContainerStarted","Data":"9887c12fc21666d17b6b1967c668cd55535655ae02d185fa01761b62abf43d6d"}
Oct 31 01:26:28.799208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.787447    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" podStartSLOduration=1.7874265459999998 podStartE2EDuration="1.787426546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:28.786528832 +0000 UTC m=+1310.087485446" watchObservedRunningTime="2026-10-31 01:26:28.787426546 +0000 UTC m=+1310.088383126"
Oct 31 01:26:28.961221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:28.960631    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:29.235840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:29.235810    5419 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:26:29.236007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:29.235860    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 01:26:30.235356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:30.235319    5419 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:26:30.235772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:30.235373    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 01:26:31.235754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:31.235723    5419 patch_prober.go:29] interesting pod/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 01:26:31.236187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:31.235783    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 01:26:32.239079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:32.239049    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:26:32.239551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:32.239476    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:26:32.240178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:32.240128    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:26:37.813599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.813565    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"]
Oct 31 01:26:37.816439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.816423    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.919029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.918971    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18bf762c\"/\"kube-root-ca.crt\""
Oct 31 01:26:37.919029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.919027    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18bf762c\"/\"e2e-lifecycle-model-18bf762c-kserve-self-signed-certs\""
Oct 31 01:26:37.919295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.919027    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18bf762c\"/\"openshift-service-ca.crt\""
Oct 31 01:26:37.919295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.919095    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18bf762c\"/\"default-dockercfg-ps9lg\""
Oct 31 01:26:37.923037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.923014    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"]
Oct 31 01:26:37.944961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.944933    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.945117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.945088    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.945218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.945164    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562n8\" (UniqueName: \"kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.945218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.945190    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.945355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.945337    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.945483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.945366    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:37.985165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:37.983329    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.047464    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.047529    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562n8\" (UniqueName: \"kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.047561    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.047725    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.047759    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.047978    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.048493    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.048728    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.053334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.050830    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.069483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.058682    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.069730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.068087    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.080436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.080407    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:38.105037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.105004    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562n8\" (UniqueName: \"kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8\") pod \"e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") " pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.128646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.128611    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:38.522068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.522024    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"]
Oct 31 01:26:38.522458 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:26:38.522437    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94b2bd66_5e24_4fea_abb7_0bf82992a828.slice/crio-8b53a255b2588558738befd87e8ac7c7e141795dbaab1381f6ffb313a2e9c735 WatchSource:0}: Error finding container 8b53a255b2588558738befd87e8ac7c7e141795dbaab1381f6ffb313a2e9c735: Status 404 returned error can't find the container with id 8b53a255b2588558738befd87e8ac7c7e141795dbaab1381f6ffb313a2e9c735
Oct 31 01:26:38.743758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.743719    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" event={"ID":"94b2bd66-5e24-4fea-abb7-0bf82992a828","Type":"ContainerStarted","Data":"525f1a95ba2c7ab15788c239fe656ac7979ee3c9178c98c08e77c6953ecee97c"}
Oct 31 01:26:38.743758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.743768    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" event={"ID":"94b2bd66-5e24-4fea-abb7-0bf82992a828","Type":"ContainerStarted","Data":"8b53a255b2588558738befd87e8ac7c7e141795dbaab1381f6ffb313a2e9c735"}
Oct 31 01:26:38.755265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.755239    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:26:38.864698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:38.862506    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" podStartSLOduration=1.862488718 podStartE2EDuration="1.862488718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:38.8590814 +0000 UTC m=+1320.160037978" watchObservedRunningTime="2026-10-31 01:26:38.862488718 +0000 UTC m=+1320.163445296"
Oct 31 01:26:39.129253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.129173    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:39.260835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.260807    5419 scope.go:122] "RemoveContainer" containerID="6a18f86b99ec85b60cbea4b6df3081f37fd87e0996a098e20ad809b9b2674062"
Oct 31 01:26:39.270652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.270629    5419 scope.go:122] "RemoveContainer" containerID="9694594dab2129ae8c5f3e64653f2637df22d63a18bfcde391a6923c03da0221"
Oct 31 01:26:39.279532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.279514    5419 scope.go:122] "RemoveContainer" containerID="f367c692a7af4a63819d0657bf48c5597b01cda33ad83a64b6d623ddb4a52ada"
Oct 31 01:26:39.288537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.288522    5419 scope.go:122] "RemoveContainer" containerID="1cd6ce17a9e3d09fedcf1e24226fa1a7fefdb5d9b94466664bbfbd4449e4fc65"
Oct 31 01:26:39.382267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.382195    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"]
Oct 31 01:26:39.386057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.386030    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.438333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.438201    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:26:39.438333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.438291    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:26:39.443038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.443011    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"]
Oct 31 01:26:39.487292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.487269    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:26:39.487448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.487303    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:26:39.499476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.499448    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.499697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.499678    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8785w\" (UniqueName: \"kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.499752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.499722    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.601162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.601104    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8785w\" (UniqueName: \"kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.601336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.601184    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.601336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.601233    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.601824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.601804    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.603801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.603776    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.658596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.658528    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8785w\" (UniqueName: \"kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w\") pod \"maas-api-revoke-sub-6e4ba8ff-tf7g2\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:39.698767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:39.698734    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:40.074310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:40.074284    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"]
Oct 31 01:26:40.074848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:26:40.074818    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59eb89eb_96df_42b7_ab07_0410e9c74c6d.slice/crio-992cb7dd7333c1a1b8311010ff9f037efe717f56e7352d090ebc228046efba93 WatchSource:0}: Error finding container 992cb7dd7333c1a1b8311010ff9f037efe717f56e7352d090ebc228046efba93: Status 404 returned error can't find the container with id 992cb7dd7333c1a1b8311010ff9f037efe717f56e7352d090ebc228046efba93
Oct 31 01:26:40.769613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:40.769573    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2" event={"ID":"59eb89eb-96df-42b7-ab07-0410e9c74c6d","Type":"ContainerStarted","Data":"3ffdcd5c708b2695c417cbcf100d5b1b80a2dcae0e9aa26d7675a73ef6b4491e"}
Oct 31 01:26:40.769805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:40.769655    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2" event={"ID":"59eb89eb-96df-42b7-ab07-0410e9c74c6d","Type":"ContainerStarted","Data":"992cb7dd7333c1a1b8311010ff9f037efe717f56e7352d090ebc228046efba93"}
Oct 31 01:26:40.882577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:40.882524    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2" podStartSLOduration=1.8825093499999999 podStartE2EDuration="1.88250935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:40.881728638 +0000 UTC m=+1322.182685229" watchObservedRunningTime="2026-10-31 01:26:40.88250935 +0000 UTC m=+1322.183465930"
Oct 31 01:26:41.780836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:41.780801    5419 generic.go:356] "Generic (PLEG): container finished" podID="59eb89eb-96df-42b7-ab07-0410e9c74c6d" containerID="3ffdcd5c708b2695c417cbcf100d5b1b80a2dcae0e9aa26d7675a73ef6b4491e" exitCode=0
Oct 31 01:26:41.781275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:41.780841    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2" event={"ID":"59eb89eb-96df-42b7-ab07-0410e9c74c6d","Type":"ContainerDied","Data":"3ffdcd5c708b2695c417cbcf100d5b1b80a2dcae0e9aa26d7675a73ef6b4491e"}
Oct 31 01:26:42.953985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:42.953962    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:43.051468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.051396    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca\") pod \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") "
Oct 31 01:26:43.051615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.051522    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w\" (UniqueName: \"kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w\") pod \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") "
Oct 31 01:26:43.051615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.051562    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca\") pod \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\" (UID: \"59eb89eb-96df-42b7-ab07-0410e9c74c6d\") "
Oct 31 01:26:43.051748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.051726    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca" pod "59eb89eb-96df-42b7-ab07-0410e9c74c6d" (UID: "59eb89eb-96df-42b7-ab07-0410e9c74c6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:43.051947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.051933    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:43.054258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.054234    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca" pod "59eb89eb-96df-42b7-ab07-0410e9c74c6d" (UID: "59eb89eb-96df-42b7-ab07-0410e9c74c6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:26:43.054345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.054331    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w" pod "59eb89eb-96df-42b7-ab07-0410e9c74c6d" (UID: "59eb89eb-96df-42b7-ab07-0410e9c74c6d"). InnerVolumeSpecName "kube-api-access-8785w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:43.153528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.153498    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8785w\" (UniqueName: \"kubernetes.io/projected/59eb89eb-96df-42b7-ab07-0410e9c74c6d-kube-api-access-8785w\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:43.153528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.153528    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59eb89eb-96df-42b7-ab07-0410e9c74c6d-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:26:43.796196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.796169    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"
Oct 31 01:26:43.796196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.796172    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2" event={"ID":"59eb89eb-96df-42b7-ab07-0410e9c74c6d","Type":"ContainerDied","Data":"992cb7dd7333c1a1b8311010ff9f037efe717f56e7352d090ebc228046efba93"}
Oct 31 01:26:43.796404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:43.796220    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="992cb7dd7333c1a1b8311010ff9f037efe717f56e7352d090ebc228046efba93"
Oct 31 01:26:44.582587 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:26:44.582548    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:26:47.090815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:47.090773    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:26:48.138309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:48.138277    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:48.317989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:48.317959    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:26:48.868951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:26:48.868922    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:27:03.480661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.480625    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"]
Oct 31 01:27:03.481210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.481196    5419 cpu_manager.go:414] "containerMap: removing container" podUID="59eb89eb-96df-42b7-ab07-0410e9c74c6d" containerName="revoke-keys"
Oct 31 01:27:03.481264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.481211    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59eb89eb-96df-42b7-ab07-0410e9c74c6d" containerName="revoke-keys"
Oct 31 01:27:03.481329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.481320    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="59eb89eb-96df-42b7-ab07-0410e9c74c6d" containerName="revoke-keys"
Oct 31 01:27:03.484874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.484857    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573052    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573103    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573232    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573302    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573357    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573392    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573417    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573471    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974fh\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.573669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.573649    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.590482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.590449    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"]
Oct 31 01:27:03.674683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674648    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.674886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674715    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.674886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674739    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.674886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674765    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.674886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674789    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.674886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674815    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974fh\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.674931    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675003    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675085    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675123    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675194    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675268    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675534    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.675603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.675560    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.678094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.678066    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.678282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.678265    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.731764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.731690    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.731901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.731806    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974fh\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:03.798296 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:03.798267    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:04.258988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:04.258958    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"]
Oct 31 01:27:04.260225 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:27:04.260195    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ea7f89_4b8c_41d4_8107_f95ad9aa6f71.slice/crio-83c183032a2c120cfd36b0bf80a076772152463ed5c573ec9c8680f449728be2 WatchSource:0}: Error finding container 83c183032a2c120cfd36b0bf80a076772152463ed5c573ec9c8680f449728be2: Status 404 returned error can't find the container with id 83c183032a2c120cfd36b0bf80a076772152463ed5c573ec9c8680f449728be2
Oct 31 01:27:04.262820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:04.262783    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:04.262935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:04.262866    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:04.262935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:04.262914    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:05.015661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:05.015616    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" event={"ID":"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71","Type":"ContainerStarted","Data":"696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0"}
Oct 31 01:27:05.016113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:05.015691    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" event={"ID":"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71","Type":"ContainerStarted","Data":"83c183032a2c120cfd36b0bf80a076772152463ed5c573ec9c8680f449728be2"}
Oct 31 01:27:05.799390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:05.799355    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:05.801205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:05.801178    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 01:27:05.804196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:05.801232    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 01:27:06.799573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:06.799534    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 01:27:06.799992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:06.799592    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 01:27:07.799586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:07.799540    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 01:27:07.800058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:07.799625    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 01:27:08.799323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:08.799285    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 01:27:08.799532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:08.799344    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 01:27:09.799505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:09.799470    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 01:27:09.800208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:09.800177    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 01:27:10.799758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:10.799719    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 01:27:10.800175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:10.799792    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 01:27:11.395929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.395876    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podStartSLOduration=8.395861998 podStartE2EDuration="8.395861998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:05.2378994 +0000 UTC m=+1346.538855976" watchObservedRunningTime="2026-10-31 01:27:11.395861998 +0000 UTC m=+1352.696818574"
Oct 31 01:27:11.399109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.399088    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"]
Oct 31 01:27:11.403110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.403089    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414127    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9gkf\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414307    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414345    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414372    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414514    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414565    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414654    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414704    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.414753 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.414725    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.507081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.507031    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"]
Oct 31 01:27:11.516060 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516024    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.516257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516102    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.516257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516182    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.516257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516238    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.516433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516276    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.516433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516344    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9gkf\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.516937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516908    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.517061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.516970    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.518812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.517402    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.518812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.518592    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.518812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.518660    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.519109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.519048    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.520119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.520094    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.520245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.518696    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.521981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.521960    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.522393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.522371    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.574649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.574620    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9gkf\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.574869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.574647    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:11.718442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:11.718363    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:12.181811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:12.181768    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"]
Oct 31 01:27:12.182362 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:27:12.181824    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19466f9_e55d_490b_a30e_734467091321.slice/crio-ccee567cebab4134d0e945a698d329ea2d841bd0215740c8827c27708c801055 WatchSource:0}: Error finding container ccee567cebab4134d0e945a698d329ea2d841bd0215740c8827c27708c801055: Status 404 returned error can't find the container with id ccee567cebab4134d0e945a698d329ea2d841bd0215740c8827c27708c801055
Oct 31 01:27:12.184339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:12.184309    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:12.184468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:12.184368    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:12.184468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:12.184401    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:12.840291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:12.840259    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:27:12.840483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:12.840310    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:13.087469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.087426    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" event={"ID":"b19466f9-e55d-490b-a30e-734467091321","Type":"ContainerStarted","Data":"75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a"}
Oct 31 01:27:13.088884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.088854    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" event={"ID":"b19466f9-e55d-490b-a30e-734467091321","Type":"ContainerStarted","Data":"ccee567cebab4134d0e945a698d329ea2d841bd0215740c8827c27708c801055"}
Oct 31 01:27:13.165762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.165688    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:13.165993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.165973    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:13.166725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.166706    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:27:13.208195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.208126    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podStartSLOduration=2.2081104639999998 podStartE2EDuration="2.208110464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:13.204542183 +0000 UTC m=+1354.505498762" watchObservedRunningTime="2026-10-31 01:27:13.208110464 +0000 UTC m=+1354.509067040"
Oct 31 01:27:13.719320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.719278    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:13.720432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.720405    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 01:27:13.720563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:13.720466    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 01:27:14.605517 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:27:14.605470    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:27:14.719752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:14.719708    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 01:27:14.719952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:14.719779    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 01:27:15.718925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:15.718887    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 01:27:15.719380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:15.718943    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 01:27:16.719780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:16.719741    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 01:27:16.720201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:16.719809    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 01:27:17.719742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:17.719622    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 01:27:17.719940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:17.719740    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 01:27:18.719724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:18.719689    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 01:27:18.719928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:18.719776    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 01:27:19.724024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:19.723994    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:19.724460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:19.724359    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:19.725093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:19.725075    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:27:20.256839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.256802    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"]
Oct 31 01:27:20.261827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.261801    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.313836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.313792    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:27:20.313836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.313825    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:27:20.314070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.313933    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:27:20.314070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.313964    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:27:20.318478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.318454    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"]
Oct 31 01:27:20.444015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.443977    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpb8d\" (UniqueName: \"kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.444228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.444065    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.444295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.444226    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.546354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.546275    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpb8d\" (UniqueName: \"kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.546354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.546340    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.546561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.546393    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.547057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.547022    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.548867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.548844    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.602547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.602509    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpb8d\" (UniqueName: \"kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d\") pod \"maas-api-revoke-sub-a1ccf031-7rgqs\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:20.876856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:20.876771    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:21.274865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:21.274833    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs" event={"ID":"c651d34c-8f2a-43fb-b7e7-0dcca9827531","Type":"ContainerStarted","Data":"88f64838cdffef8255ebe4f73777b69d7f94cc7e2ab08c73aaad0462b8fc0c85"}
Oct 31 01:27:21.327048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:21.327014    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"]
Oct 31 01:27:22.283483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.283451    5419 generic.go:356] "Generic (PLEG): container finished" podID="c651d34c-8f2a-43fb-b7e7-0dcca9827531" containerID="0c68b0c31d02a5551737c152ac69cc07605f862f909fd6980c07dbdac2825bf4" exitCode=0
Oct 31 01:27:22.283859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.283534    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs" event={"ID":"c651d34c-8f2a-43fb-b7e7-0dcca9827531","Type":"ContainerDied","Data":"0c68b0c31d02a5551737c152ac69cc07605f862f909fd6980c07dbdac2825bf4"}
Oct 31 01:27:22.702475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.702447    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"]
Oct 31 01:27:22.706469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.706452    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.762728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.762696    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"]
Oct 31 01:27:22.779320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.779289    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.779513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.779490    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9pp2\" (UniqueName: \"kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.779676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.779658    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.881043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.881014    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9pp2\" (UniqueName: \"kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.881231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.881131    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.881231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.881209    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.881905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.881884    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.883729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.883704    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:22.937790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:22.937764    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9pp2\" (UniqueName: \"kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2\") pod \"maas-api-revoke-sub-7ac23bbc-p2qrp\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:23.020201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.020171    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:23.396336 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:27:23.396285    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb32303_d237_442c_89bb_e0c5584615f0.slice/crio-986a534c7dd7fda94afa99aec4397ccb090385a7c83caa5b0754e11031ea4cab WatchSource:0}: Error finding container 986a534c7dd7fda94afa99aec4397ccb090385a7c83caa5b0754e11031ea4cab: Status 404 returned error can't find the container with id 986a534c7dd7fda94afa99aec4397ccb090385a7c83caa5b0754e11031ea4cab
Oct 31 01:27:23.397612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.397586    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"]
Oct 31 01:27:23.427250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.427227    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:23.488739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.488708    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d\" (UniqueName: \"kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d\") pod \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") "
Oct 31 01:27:23.488739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.488741    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca\") pod \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") "
Oct 31 01:27:23.488954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.488789    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca\") pod \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\" (UID: \"c651d34c-8f2a-43fb-b7e7-0dcca9827531\") "
Oct 31 01:27:23.489222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.489198    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca" pod "c651d34c-8f2a-43fb-b7e7-0dcca9827531" (UID: "c651d34c-8f2a-43fb-b7e7-0dcca9827531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:27:23.490860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.490841    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d" pod "c651d34c-8f2a-43fb-b7e7-0dcca9827531" (UID: "c651d34c-8f2a-43fb-b7e7-0dcca9827531"). InnerVolumeSpecName "kube-api-access-cpb8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:23.490929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.490876    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca" pod "c651d34c-8f2a-43fb-b7e7-0dcca9827531" (UID: "c651d34c-8f2a-43fb-b7e7-0dcca9827531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:27:23.590750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.590477    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpb8d\" (UniqueName: \"kubernetes.io/projected/c651d34c-8f2a-43fb-b7e7-0dcca9827531-kube-api-access-cpb8d\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:23.590750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.590502    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:23.590750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:23.590512    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c651d34c-8f2a-43fb-b7e7-0dcca9827531-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:24.298996 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:24.298961    5419 generic.go:356] "Generic (PLEG): container finished" podID="1cb32303-d237-442c-89bb-e0c5584615f0" containerID="b2ab9f9e067dcf766cf70f6210001d98cb67b9eec007f822d32fe5b1d04afb7b" exitCode=0
Oct 31 01:27:24.299195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:24.299047    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp" event={"ID":"1cb32303-d237-442c-89bb-e0c5584615f0","Type":"ContainerDied","Data":"b2ab9f9e067dcf766cf70f6210001d98cb67b9eec007f822d32fe5b1d04afb7b"}
Oct 31 01:27:24.299195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:24.299103    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp" event={"ID":"1cb32303-d237-442c-89bb-e0c5584615f0","Type":"ContainerStarted","Data":"986a534c7dd7fda94afa99aec4397ccb090385a7c83caa5b0754e11031ea4cab"}
Oct 31 01:27:24.300472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:24.300455    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"
Oct 31 01:27:24.300515 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:24.300488    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs" event={"ID":"c651d34c-8f2a-43fb-b7e7-0dcca9827531","Type":"ContainerDied","Data":"88f64838cdffef8255ebe4f73777b69d7f94cc7e2ab08c73aaad0462b8fc0c85"}
Oct 31 01:27:24.300554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:24.300524    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88f64838cdffef8255ebe4f73777b69d7f94cc7e2ab08c73aaad0462b8fc0c85"
Oct 31 01:27:25.140699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.140666    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:27:25.464801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.464770    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:25.529200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.529172    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2\" (UniqueName: \"kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2\") pod \"1cb32303-d237-442c-89bb-e0c5584615f0\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") "
Oct 31 01:27:25.529380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.529275    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca\") pod \"1cb32303-d237-442c-89bb-e0c5584615f0\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") "
Oct 31 01:27:25.529380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.529321    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca\") pod \"1cb32303-d237-442c-89bb-e0c5584615f0\" (UID: \"1cb32303-d237-442c-89bb-e0c5584615f0\") "
Oct 31 01:27:25.529661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.529630    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca" pod "1cb32303-d237-442c-89bb-e0c5584615f0" (UID: "1cb32303-d237-442c-89bb-e0c5584615f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:27:25.529877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.529858    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:25.531590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.531558    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca" pod "1cb32303-d237-442c-89bb-e0c5584615f0" (UID: "1cb32303-d237-442c-89bb-e0c5584615f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:27:25.531840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.531820    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2" pod "1cb32303-d237-442c-89bb-e0c5584615f0" (UID: "1cb32303-d237-442c-89bb-e0c5584615f0"). InnerVolumeSpecName "kube-api-access-k9pp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:25.665317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.661245    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9pp2\" (UniqueName: \"kubernetes.io/projected/1cb32303-d237-442c-89bb-e0c5584615f0-kube-api-access-k9pp2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:25.665317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:25.661287    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cb32303-d237-442c-89bb-e0c5584615f0-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:26.317693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:26.317660    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"
Oct 31 01:27:26.318182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:26.317683    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp" event={"ID":"1cb32303-d237-442c-89bb-e0c5584615f0","Type":"ContainerDied","Data":"986a534c7dd7fda94afa99aec4397ccb090385a7c83caa5b0754e11031ea4cab"}
Oct 31 01:27:26.318182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:26.317760    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="986a534c7dd7fda94afa99aec4397ccb090385a7c83caa5b0754e11031ea4cab"
Oct 31 01:27:33.565525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.565490    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"]
Oct 31 01:27:33.566067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.566051    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c651d34c-8f2a-43fb-b7e7-0dcca9827531" containerName="revoke-keys"
Oct 31 01:27:33.566112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.566068    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c651d34c-8f2a-43fb-b7e7-0dcca9827531" containerName="revoke-keys"
Oct 31 01:27:33.566112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.566090    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1cb32303-d237-442c-89bb-e0c5584615f0" containerName="revoke-keys"
Oct 31 01:27:33.566112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.566096    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb32303-d237-442c-89bb-e0c5584615f0" containerName="revoke-keys"
Oct 31 01:27:33.566253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.566241    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb32303-d237-442c-89bb-e0c5584615f0" containerName="revoke-keys"
Oct 31 01:27:33.566290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.566259    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c651d34c-8f2a-43fb-b7e7-0dcca9827531" containerName="revoke-keys"
Oct 31 01:27:33.569713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.569696    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.627431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.627404    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"]
Oct 31 01:27:33.670964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.670930    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:27:33.670964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.670930    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:27:33.671215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.671002    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:27:33.671215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.671038    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:27:33.705797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.705770    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.705960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.705912    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.706028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.705957    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdsw9\" (UniqueName: \"kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.807752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.807710    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.807960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.807767    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdsw9\" (UniqueName: \"kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.807960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.807924    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.808539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.808502    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.810194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.810166    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.867914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.867836    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdsw9\" (UniqueName: \"kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9\") pod \"maas-api-revoke-sub-ecc7c35d-5ztdz\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:33.888430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:33.888392    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:34.310787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:34.310754    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"]
Oct 31 01:27:34.311224 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:27:34.311199    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a09efb7_fa0e_4b24_936a_86adaebd9088.slice/crio-7525c7186c914c166a96c1f656d30af28edb1af9356e9af01319e94c193973d5 WatchSource:0}: Error finding container 7525c7186c914c166a96c1f656d30af28edb1af9356e9af01319e94c193973d5: Status 404 returned error can't find the container with id 7525c7186c914c166a96c1f656d30af28edb1af9356e9af01319e94c193973d5
Oct 31 01:27:34.375919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:34.375884    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz" event={"ID":"1a09efb7-fa0e-4b24-936a-86adaebd9088","Type":"ContainerStarted","Data":"7525c7186c914c166a96c1f656d30af28edb1af9356e9af01319e94c193973d5"}
Oct 31 01:27:35.475339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:35.475302    5419 generic.go:356] "Generic (PLEG): container finished" podID="1a09efb7-fa0e-4b24-936a-86adaebd9088" containerID="5fe6e921de9f8b8fef0eabef708b45bcac9f81aacd12cbc9276347141502f40b" exitCode=0
Oct 31 01:27:35.476044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:35.476022    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz" event={"ID":"1a09efb7-fa0e-4b24-936a-86adaebd9088","Type":"ContainerDied","Data":"5fe6e921de9f8b8fef0eabef708b45bcac9f81aacd12cbc9276347141502f40b"}
Oct 31 01:27:36.715166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.714702    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:36.898355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.898277    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca\") pod \"1a09efb7-fa0e-4b24-936a-86adaebd9088\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") "
Oct 31 01:27:36.898355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.898344    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca\") pod \"1a09efb7-fa0e-4b24-936a-86adaebd9088\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") "
Oct 31 01:27:36.898561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.898437    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9\" (UniqueName: \"kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9\") pod \"1a09efb7-fa0e-4b24-936a-86adaebd9088\" (UID: \"1a09efb7-fa0e-4b24-936a-86adaebd9088\") "
Oct 31 01:27:36.898689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.898660    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca" pod "1a09efb7-fa0e-4b24-936a-86adaebd9088" (UID: "1a09efb7-fa0e-4b24-936a-86adaebd9088"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:27:36.898903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.898885    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:36.900528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.900506    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca" pod "1a09efb7-fa0e-4b24-936a-86adaebd9088" (UID: "1a09efb7-fa0e-4b24-936a-86adaebd9088"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:27:36.900674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:36.900653    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9" pod "1a09efb7-fa0e-4b24-936a-86adaebd9088" (UID: "1a09efb7-fa0e-4b24-936a-86adaebd9088"). InnerVolumeSpecName "kube-api-access-zdsw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:27:37.000120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:37.000084    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a09efb7-fa0e-4b24-936a-86adaebd9088-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:37.000120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:37.000115    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdsw9\" (UniqueName: \"kubernetes.io/projected/1a09efb7-fa0e-4b24-936a-86adaebd9088-kube-api-access-zdsw9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:27:37.490766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:37.490736    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"
Oct 31 01:27:37.490766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:37.490754    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz" event={"ID":"1a09efb7-fa0e-4b24-936a-86adaebd9088","Type":"ContainerDied","Data":"7525c7186c914c166a96c1f656d30af28edb1af9356e9af01319e94c193973d5"}
Oct 31 01:27:37.490986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:37.490813    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7525c7186c914c166a96c1f656d30af28edb1af9356e9af01319e94c193973d5"
Oct 31 01:27:44.767114 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:27:44.767059    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:27:45.763266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:45.763220    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main" probeResult="failure" output="Get \"https://10.133.0.113:8000/ready\": context deadline exceeded"
Oct 31 01:27:45.867086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:45.867041    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:45.925489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:45.925445    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 01:27:46.099605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.099481    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc" podUID="222d5179-79a4-4cfd-b925-6a899bb608ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.164330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.164277    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.464106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.463225    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.532165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.526230    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.559193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.558308    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Oct 31 01:27:46.788164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.765567    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main" probeResult="failure" output="Get \"https://10.133.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.788830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.788798    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:27:46.872013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.871973    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:27:46.883422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.883392    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:27:46.902076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.902048    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 01:27:46.956103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:46.956079    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:27:47.249038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.249004    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"]
Oct 31 01:27:47.249681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.249660    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1a09efb7-fa0e-4b24-936a-86adaebd9088" containerName="revoke-keys"
Oct 31 01:27:47.249791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.249682    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a09efb7-fa0e-4b24-936a-86adaebd9088" containerName="revoke-keys"
Oct 31 01:27:47.249860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.249847    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a09efb7-fa0e-4b24-936a-86adaebd9088" containerName="revoke-keys"
Oct 31 01:27:47.254796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.254769    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.321920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.321661    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"]
Oct 31 01:27:47.345720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.345687    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.345883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.345803    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.345945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.345927    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.345988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.345972    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.346093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.346075    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.346285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.346266    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9l8k\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.346348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.346325    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.346499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.346479    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.346561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.346544    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.357868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.357839    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-01cbd8-openshift-default-dockercfg-cg8gv\""
Oct 31 01:27:47.447738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.447706    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.447931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.447773    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.447931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.447813    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.447931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.447859    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.447930    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.447966    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448021    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448125    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9l8k\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448192    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448352    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448389    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448408    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.448574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.448564    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.449343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.449318    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.450448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.450428    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.450855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.450837    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.554225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.552237    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9l8k\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.586175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.582949    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token\") pod \"e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") " pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:47.871750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:47.871355    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:48.048417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.048384    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"]
Oct 31 01:27:48.051972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.051951    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7ff7ac-kk2w2"]
Oct 31 01:27:48.316165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.316124    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"]
Oct 31 01:27:48.317556 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:27:48.317531    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ab9da4_de4b_45a1_a08a_06f7def7d064.slice/crio-cce418fe9fbedc2f14c29a4ed5b6b57ba5fbf78ce06750a726dc7e0a2e258315 WatchSource:0}: Error finding container cce418fe9fbedc2f14c29a4ed5b6b57ba5fbf78ce06750a726dc7e0a2e258315: Status 404 returned error can't find the container with id cce418fe9fbedc2f14c29a4ed5b6b57ba5fbf78ce06750a726dc7e0a2e258315
Oct 31 01:27:48.319958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.319928    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:48.320027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.319990    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:48.320027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.320020    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:48.671031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.670951    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" event={"ID":"f0ab9da4-de4b-45a1-a08a-06f7def7d064","Type":"ContainerStarted","Data":"6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"}
Oct 31 01:27:48.671031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.671015    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" event={"ID":"f0ab9da4-de4b-45a1-a08a-06f7def7d064","Type":"ContainerStarted","Data":"cce418fe9fbedc2f14c29a4ed5b6b57ba5fbf78ce06750a726dc7e0a2e258315"}
Oct 31 01:27:48.792039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.791981    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podStartSLOduration=1.791966722 podStartE2EDuration="1.791966722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:48.788954638 +0000 UTC m=+1390.089911242" watchObservedRunningTime="2026-10-31 01:27:48.791966722 +0000 UTC m=+1390.092923353"
Oct 31 01:27:48.839012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.838512    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9391ce64-0dd5-46d0-a7ec-69ca16ece995" path="/var/lib/kubelet/pods/9391ce64-0dd5-46d0-a7ec-69ca16ece995/volumes"
Oct 31 01:27:48.843870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.843845    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"]
Oct 31 01:27:48.848206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.848190    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.872255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.872227    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948237    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948407    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948509    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948552    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948581    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948739    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948765    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948790    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.950223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.948813    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmr7q\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:48.997462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:48.997430    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"]
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.050364    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.051772    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.051815    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.051845    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.051869    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmr7q\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.051976    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.052071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.052214    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.053712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.052254    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.054762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.054585    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.055185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.055159    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.055970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.055938    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.056072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.050909    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.056500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.056471    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.059006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.058985    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.059192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.058997    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.112319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.112292    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.112454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.112437    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmr7q\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.163246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.163218    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:49.591488 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:27:49.591459    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44131ca_439b_431e_86b0_88edb2b5e9da.slice/crio-8f6a45c1d586734cbd4a14054926b9bd3f5f883d1345c4046a92aeeae714359b WatchSource:0}: Error finding container 8f6a45c1d586734cbd4a14054926b9bd3f5f883d1345c4046a92aeeae714359b: Status 404 returned error can't find the container with id 8f6a45c1d586734cbd4a14054926b9bd3f5f883d1345c4046a92aeeae714359b
Oct 31 01:27:49.591864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.591843    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"]
Oct 31 01:27:49.594602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.594569    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:49.594700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.594639    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:49.594700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.594664    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:27:49.679716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.679680    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" event={"ID":"a44131ca-439b-431e-86b0-88edb2b5e9da","Type":"ContainerStarted","Data":"c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19"}
Oct 31 01:27:49.679902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.679745    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" event={"ID":"a44131ca-439b-431e-86b0-88edb2b5e9da","Type":"ContainerStarted","Data":"8f6a45c1d586734cbd4a14054926b9bd3f5f883d1345c4046a92aeeae714359b"}
Oct 31 01:27:49.800381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.800326    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podStartSLOduration=1.8003120479999999 podStartE2EDuration="1.800312048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:27:49.797699397 +0000 UTC m=+1391.098655974" watchObservedRunningTime="2026-10-31 01:27:49.800312048 +0000 UTC m=+1391.101268704"
Oct 31 01:27:49.878298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:49.878211    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:50.164273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:50.164188    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:50.165879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:50.165860    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:50.165947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:50.165911    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:50.711125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:50.711084    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:50.714643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:50.714616    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:27:51.163839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:51.163803    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:51.163995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:51.163856    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:52.163742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:52.163705    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:52.164156 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:52.163762    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:53.164424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:53.164392    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:53.164830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:53.164444    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:54.164164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:54.164093    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:54.164358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:54.164167    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:55.168165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:55.164635    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:55.168165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:55.164756    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:56.163925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:56.163887    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 31 01:27:56.164099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:56.163962    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 31 01:27:57.182877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:57.182574    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:57.194651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:57.188767    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:57.194651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:57.192097    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:27:58.903156 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:27:58.903110    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:28:03.950731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:03.950698    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"]
Oct 31 01:28:03.955228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:03.955211    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.031866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.031831    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.031905    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.031939    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.032060    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.032093    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.032195    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.032226    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.032317    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvclc\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.032478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.032430    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.061232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.061199    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"]
Oct 31 01:28:04.133562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133529    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvclc\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.133736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133624    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.133736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133687    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.133736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133727    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.133864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133753    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.133864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133817    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.133864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133850    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133901    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.133949    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.134111    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.134190    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.134287    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.134390    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.134563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.134542    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.136003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.135985    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.136476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.136455    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.189854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.189820    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.190002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.189824    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvclc\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.273005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.272716    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:04.746600 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:28:04.746569    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae37bc44_518e_4d47_8dc9_a3a493656aaf.slice/crio-60a4ec9404802c3f5e1f81c4cc1c36f4410e36d629929f9cbdbfe57a12a1e799 WatchSource:0}: Error finding container 60a4ec9404802c3f5e1f81c4cc1c36f4410e36d629929f9cbdbfe57a12a1e799: Status 404 returned error can't find the container with id 60a4ec9404802c3f5e1f81c4cc1c36f4410e36d629929f9cbdbfe57a12a1e799
Oct 31 01:28:04.746923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.746900    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"]
Oct 31 01:28:04.749012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.748983    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:28:04.749092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.749046    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:28:04.749092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.749077    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:28:04.975595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.975562    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" event={"ID":"ae37bc44-518e-4d47-8dc9-a3a493656aaf","Type":"ContainerStarted","Data":"670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601"}
Oct 31 01:28:04.975999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:04.975614    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" event={"ID":"ae37bc44-518e-4d47-8dc9-a3a493656aaf","Type":"ContainerStarted","Data":"60a4ec9404802c3f5e1f81c4cc1c36f4410e36d629929f9cbdbfe57a12a1e799"}
Oct 31 01:28:05.045696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:05.045601    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"]
Oct 31 01:28:05.049121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:05.049091    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-384310bc-rhk2p"]
Oct 31 01:28:05.097559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:05.097503    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podStartSLOduration=2.097485181 podStartE2EDuration="2.097485181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:05.093305609 +0000 UTC m=+1406.394262186" watchObservedRunningTime="2026-10-31 01:28:05.097485181 +0000 UTC m=+1406.398441761"
Oct 31 01:28:05.273840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:05.273808    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:05.275254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:05.275228    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:05.275375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:05.275278    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:06.293656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:06.293615    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:06.294418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:06.294381    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:06.843101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:06.843063    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a093b4d2-d792-4541-a260-c40380c86fa1" path="/var/lib/kubelet/pods/a093b4d2-d792-4541-a260-c40380c86fa1/volumes"
Oct 31 01:28:07.274290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:07.274253    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:07.274575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:07.274543    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:08.273694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:08.273661    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:08.274086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:08.273729    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:09.297166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:09.285773    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:09.297166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:09.285849    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:10.273831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:10.273792    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:10.274043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:10.273853    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:11.274056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:11.274020    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Oct 31 01:28:11.274451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:11.274077    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Oct 31 01:28:12.679885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:12.679852    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:12.680567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:12.680545    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:12.681617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:12.681597    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:28:14.583703 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:28:14.583660    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:28:21.048915 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:21.048881    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"]
Oct 31 01:28:21.098128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:21.098096    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c85b3b9-78hvx"]
Oct 31 01:28:22.838187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:22.838127    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f3c7c8-7fd6-4518-b408-e0208f6a6a84" path="/var/lib/kubelet/pods/e8f3c7c8-7fd6-4518-b408-e0208f6a6a84/volumes"
Oct 31 01:28:26.714762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.714715    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"]
Oct 31 01:28:26.730166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.727350    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.822477    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.822559    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.822623    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.822823    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.822879    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.822936    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.823107    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.823187    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.826192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.823274    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47cv\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.939513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.939483    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"]
Oct 31 01:28:26.941226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941189    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941248    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941306    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941482    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941534    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941567    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941673    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941705    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.941784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.941758    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47cv\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.943562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.943528    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.971209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.951257    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.971209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.951969    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.971209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.952004    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.971209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.953406    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.971209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.965925    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:26.978368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:26.977963    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:27.006088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.006054    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:27.020160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.010994    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47cv\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:27.065690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.065643    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:27.565259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.564195    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"]
Oct 31 01:28:27.600187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.598189    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:28:27.600187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.598284    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:28:27.600187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:27.598325    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:28:28.248343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:28.248309    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" event={"ID":"b45bb8c6-dd67-4747-92d6-65f9522ed786","Type":"ContainerStarted","Data":"5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459"}
Oct 31 01:28:28.248745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:28.248359    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" event={"ID":"b45bb8c6-dd67-4747-92d6-65f9522ed786","Type":"ContainerStarted","Data":"61dfd67436121cd81582befbd4f7a0746cde97827dfb2af778878c37423293b6"}
Oct 31 01:28:29.066038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:29.065989    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:29.067078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:29.067045    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:28:29.067266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:29.067108    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:28:30.065990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:30.065956    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:28:30.066418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:30.066011    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:28:31.066562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:31.066523    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:28:31.066950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:31.066598    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:28:32.066985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:32.066950    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:28:32.067505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:32.067023    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:28:33.068167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:33.067995    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:28:33.068167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:33.068061    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:28:34.066825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:34.066792    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 31 01:28:34.067002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:34.066845    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 31 01:28:35.048344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:35.048283    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podStartSLOduration=9.048261506 podStartE2EDuration="9.048261506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:28:28.397262438 +0000 UTC m=+1429.698219016" watchObservedRunningTime="2026-10-31 01:28:35.048261506 +0000 UTC m=+1436.349218084"
Oct 31 01:28:35.048917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:35.048896    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"]
Oct 31 01:28:35.070841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:35.070813    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:35.071210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:35.071186    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:35.071860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:35.071838    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:28:35.095786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:35.095764    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5bbcb6-fzzqk"]
Oct 31 01:28:36.838908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:36.838875    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af491e9-c2d4-437a-a697-1026757f9373" path="/var/lib/kubelet/pods/9af491e9-c2d4-437a-a697-1026757f9373/volumes"
Oct 31 01:28:44.583508 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:28:44.583472    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:28:45.046834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:45.046803    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"]
Oct 31 01:28:45.095332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:45.095298    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6c9ea7-4lb79"]
Oct 31 01:28:46.839040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:46.839011    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf22a86-37a0-48c1-977d-1bea32f27b2a" path="/var/lib/kubelet/pods/eaf22a86-37a0-48c1-977d-1bea32f27b2a/volumes"
Oct 31 01:28:47.048423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:47.048395    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"]
Oct 31 01:28:47.097607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:47.097534    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53e9b01-nxwxt"]
Oct 31 01:28:48.838223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:48.838182    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d2da4f-c02d-41e7-8d63-48b82642881a" path="/var/lib/kubelet/pods/95d2da4f-c02d-41e7-8d63-48b82642881a/volumes"
Oct 31 01:28:49.046936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:49.046902    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"]
Oct 31 01:28:49.097251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:49.097178    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43893bca-hmckf"]
Oct 31 01:28:50.838413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:50.838382    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe463af-30de-445f-8c84-464992ae3049" path="/var/lib/kubelet/pods/2fe463af-30de-445f-8c84-464992ae3049/volumes"
Oct 31 01:28:52.496194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:52.493087    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:28:58.045861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:58.045827    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"]
Oct 31 01:28:58.094897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:58.094862    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add3ec25-xj65h"]
Oct 31 01:28:58.838328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:28:58.838296    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f" path="/var/lib/kubelet/pods/b7d6c7d2-e25d-4d5f-818b-2cac7c35b18f/volumes"
Oct 31 01:29:00.682837 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.682802    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"]
Oct 31 01:29:00.689551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.689517    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.787160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.785511    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c70e7-openshift-default-dockercfg-dl2p6\""
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.847669    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.847738    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.847802    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kkpq\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.847858    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.847900    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.848045    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.848084    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.848163    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.851162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.848201    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.948357    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"]
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949541    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949592    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949656    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kkpq\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949713    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949755    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949907    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.949948    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.950010    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.950046    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.950747    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.951073    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.951579    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.972911    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.973627    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:00.977581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:00.975274    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:01.044977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.044843    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kkpq\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:01.048302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.048233    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:01.083591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.081755    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo\") pod \"e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") " pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:01.312363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.312272    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:01.781473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.781438    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"]
Oct 31 01:29:01.782784 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:29:01.782761    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1503f6f8_f1f6_4b40_a806_0657fc0da67b.slice/crio-4c5de83293c168174b6d15ee64c110ddaac4c9b40822aff8dafeb6d6c3492e39 WatchSource:0}: Error finding container 4c5de83293c168174b6d15ee64c110ddaac4c9b40822aff8dafeb6d6c3492e39: Status 404 returned error can't find the container with id 4c5de83293c168174b6d15ee64c110ddaac4c9b40822aff8dafeb6d6c3492e39
Oct 31 01:29:01.785411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.785380    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:01.785504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.785448    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:01.785504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:01.785478    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:02.194844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.194756    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"]
Oct 31 01:29:02.236501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.225465    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.236825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.228363    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"]
Oct 31 01:29:02.301724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.301634    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:29:02.302407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.302387    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:29:02.315954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.304324    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:29:02.315954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.313292    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:29:02.315954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.314206    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.318495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.318023    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8n96\" (UniqueName: \"kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.318495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.318111    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.435677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.430280    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.435677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.430524    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8n96\" (UniqueName: \"kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.435677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.430604    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.435677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.432052    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.454566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.454049    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.516228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.514980    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8n96\" (UniqueName: \"kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96\") pod \"maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.594436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.594396    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:02.707918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.707829    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" event={"ID":"1503f6f8-f1f6-4b40-a806-0657fc0da67b","Type":"ContainerStarted","Data":"85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"}
Oct 31 01:29:02.707918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.707898    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" event={"ID":"1503f6f8-f1f6-4b40-a806-0657fc0da67b","Type":"ContainerStarted","Data":"4c5de83293c168174b6d15ee64c110ddaac4c9b40822aff8dafeb6d6c3492e39"}
Oct 31 01:29:02.828630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:02.828580    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podStartSLOduration=2.828565323 podStartE2EDuration="2.828565323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:02.826475256 +0000 UTC m=+1464.127431833" watchObservedRunningTime="2026-10-31 01:29:02.828565323 +0000 UTC m=+1464.129521900"
Oct 31 01:29:03.087793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.087743    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"]
Oct 31 01:29:03.125392 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:29:03.119574    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e693c0e_ea2d_4860_b2bb_6de8308d0c82.slice/crio-2ca0e1f7bdc6e9939361b063f26973e093173f855b710eaa7a3555f3d1bb7024 WatchSource:0}: Error finding container 2ca0e1f7bdc6e9939361b063f26973e093173f855b710eaa7a3555f3d1bb7024: Status 404 returned error can't find the container with id 2ca0e1f7bdc6e9939361b063f26973e093173f855b710eaa7a3555f3d1bb7024
Oct 31 01:29:03.317879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.317847    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:03.323493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.323467    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:03.717525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.717438    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4" event={"ID":"2e693c0e-ea2d-4860-b2bb-6de8308d0c82","Type":"ContainerStarted","Data":"69684c0660c111545130688d9f4fe519bf53c14c83cbd8054bfa69ebbe6fb652"}
Oct 31 01:29:03.717525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.717499    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4" event={"ID":"2e693c0e-ea2d-4860-b2bb-6de8308d0c82","Type":"ContainerStarted","Data":"2ca0e1f7bdc6e9939361b063f26973e093173f855b710eaa7a3555f3d1bb7024"}
Oct 31 01:29:03.717825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.717802    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:03.718846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.718827    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:29:03.967321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:03.967269    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4" podStartSLOduration=2.96725386 podStartE2EDuration="2.96725386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:03.840741666 +0000 UTC m=+1465.141698243" watchObservedRunningTime="2026-10-31 01:29:03.96725386 +0000 UTC m=+1465.268210437"
Oct 31 01:29:04.096007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:04.095973    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"]
Oct 31 01:29:04.103929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:04.102940    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f98a027-22qzp"]
Oct 31 01:29:04.727423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:04.727387    5419 generic.go:356] "Generic (PLEG): container finished" podID="2e693c0e-ea2d-4860-b2bb-6de8308d0c82" containerID="69684c0660c111545130688d9f4fe519bf53c14c83cbd8054bfa69ebbe6fb652" exitCode=0
Oct 31 01:29:04.727605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:04.727485    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4" event={"ID":"2e693c0e-ea2d-4860-b2bb-6de8308d0c82","Type":"ContainerDied","Data":"69684c0660c111545130688d9f4fe519bf53c14c83cbd8054bfa69ebbe6fb652"}
Oct 31 01:29:04.838943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:04.838915    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd741700-44e7-4767-ba82-cb78ffe52349" path="/var/lib/kubelet/pods/fd741700-44e7-4767-ba82-cb78ffe52349/volumes"
Oct 31 01:29:05.901942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:05.901913    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:06.046086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.046014    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca\") pod \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") "
Oct 31 01:29:06.046299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.046152    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca\") pod \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") "
Oct 31 01:29:06.046374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.046312    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96\" (UniqueName: \"kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96\") pod \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\" (UID: \"2e693c0e-ea2d-4860-b2bb-6de8308d0c82\") "
Oct 31 01:29:06.046429 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.046393    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca" pod "2e693c0e-ea2d-4860-b2bb-6de8308d0c82" (UID: "2e693c0e-ea2d-4860-b2bb-6de8308d0c82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:06.046913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.046889    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:06.048567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.048539    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca" pod "2e693c0e-ea2d-4860-b2bb-6de8308d0c82" (UID: "2e693c0e-ea2d-4860-b2bb-6de8308d0c82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:29:06.048672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.048589    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96" pod "2e693c0e-ea2d-4860-b2bb-6de8308d0c82" (UID: "2e693c0e-ea2d-4860-b2bb-6de8308d0c82"). InnerVolumeSpecName "kube-api-access-c8n96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:06.070766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.070734    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"]
Oct 31 01:29:06.075018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.074996    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c7fbb2f-q75jn"]
Oct 31 01:29:06.148253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.148225    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8n96\" (UniqueName: \"kubernetes.io/projected/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-kube-api-access-c8n96\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:06.148253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.148248    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e693c0e-ea2d-4860-b2bb-6de8308d0c82-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:06.742809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.742780    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"
Oct 31 01:29:06.742809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.742801    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4" event={"ID":"2e693c0e-ea2d-4860-b2bb-6de8308d0c82","Type":"ContainerDied","Data":"2ca0e1f7bdc6e9939361b063f26973e093173f855b710eaa7a3555f3d1bb7024"}
Oct 31 01:29:06.743091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.742887    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ca0e1f7bdc6e9939361b063f26973e093173f855b710eaa7a3555f3d1bb7024"
Oct 31 01:29:06.838512 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:06.838482    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5830bedb-a59e-4144-855d-92e5a6fa2e09" path="/var/lib/kubelet/pods/5830bedb-a59e-4144-855d-92e5a6fa2e09/volumes"
Oct 31 01:29:10.335538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.335498    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"]
Oct 31 01:29:10.336316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.336298    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2e693c0e-ea2d-4860-b2bb-6de8308d0c82" containerName="revoke-keys"
Oct 31 01:29:10.336372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.336317    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e693c0e-ea2d-4860-b2bb-6de8308d0c82" containerName="revoke-keys"
Oct 31 01:29:10.336440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.336430    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e693c0e-ea2d-4860-b2bb-6de8308d0c82" containerName="revoke-keys"
Oct 31 01:29:10.341694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.341669    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.388794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.388755    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"]
Oct 31 01:29:10.432160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.431547    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-42c96d\""
Oct 31 01:29:10.443637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.432214    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-42c96d-dockercfg-lgr5w\""
Oct 31 01:29:10.536939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.536218    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") " pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.536939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.536325    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmr9\" (UniqueName: \"kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9\") pod \"payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") " pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.675257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.658653    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") " pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.678199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.675642    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmr9\" (UniqueName: \"kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9\") pod \"payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") " pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.680299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.675122    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") " pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.734963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.734936    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmr9\" (UniqueName: \"kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9\") pod \"payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") " pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:10.840009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:10.839984    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:11.215771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:11.215746    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"]
Oct 31 01:29:11.216300 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:29:11.216275    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f214e9f_cd7c_4497_bb9e_774a7d50827f.slice/crio-c3edc21805fe8a47673c9c95cf13bcc3f8fbdcc360978f3770f5ff6535bf04c9 WatchSource:0}: Error finding container c3edc21805fe8a47673c9c95cf13bcc3f8fbdcc360978f3770f5ff6535bf04c9: Status 404 returned error can't find the container with id c3edc21805fe8a47673c9c95cf13bcc3f8fbdcc360978f3770f5ff6535bf04c9
Oct 31 01:29:11.787217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:11.787185    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc" event={"ID":"6f214e9f-cd7c-4497-bb9e-774a7d50827f","Type":"ContainerStarted","Data":"e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c"}
Oct 31 01:29:11.787631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:11.787241    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc" event={"ID":"6f214e9f-cd7c-4497-bb9e-774a7d50827f","Type":"ContainerStarted","Data":"c3edc21805fe8a47673c9c95cf13bcc3f8fbdcc360978f3770f5ff6535bf04c9"}
Oct 31 01:29:11.787631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:11.787267    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:14.766026 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:29:14.765989    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:29:15.046218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:15.045922    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc" podStartSLOduration=5.045891544 podStartE2EDuration="5.045891544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:11.925208167 +0000 UTC m=+1473.226164746" watchObservedRunningTime="2026-10-31 01:29:15.045891544 +0000 UTC m=+1476.346848122"
Oct 31 01:29:15.078184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:15.052194    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"]
Oct 31 01:29:15.104227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:15.104194    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbc69b53-gbc4h"]
Oct 31 01:29:16.838129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:16.838094    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ac7064-20e8-4af6-90b0-a146b56e156c" path="/var/lib/kubelet/pods/91ac7064-20e8-4af6-90b0-a146b56e156c/volumes"
Oct 31 01:29:22.280454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.280416    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"]
Oct 31 01:29:22.292585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.292546    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"]
Oct 31 01:29:22.292819 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.292663    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.393649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.393598    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:29:22.393840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.393674    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:29:22.394183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.394130    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:29:22.394436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.394423    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:29:22.566270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.556709    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.566596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.566562    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8hw\" (UniqueName: \"kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.566905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.566886    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.694642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.694611    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.695844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.695707    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.696923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.696395    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.697123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.697071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8hw\" (UniqueName: \"kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.712441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.712373    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.754509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.754443    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8hw\" (UniqueName: \"kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw\") pod \"maas-api-revoke-sub-6b3a06cb-gnqk8\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:22.774649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:22.774401    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:23.069565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:23.069534    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:29:23.709050 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:29:23.709023    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf46c21_01a3_4114_b531_cf64bfe7d7c2.slice/crio-a9feb05e923516a47675f82c66e66a24772a72cdb418816e72fa50c42c433f56 WatchSource:0}: Error finding container a9feb05e923516a47675f82c66e66a24772a72cdb418816e72fa50c42c433f56: Status 404 returned error can't find the container with id a9feb05e923516a47675f82c66e66a24772a72cdb418816e72fa50c42c433f56
Oct 31 01:29:23.709374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:23.709342    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"]
Oct 31 01:29:24.277214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:24.277164    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8" event={"ID":"8bf46c21-01a3-4114-b531-cf64bfe7d7c2","Type":"ContainerStarted","Data":"a9feb05e923516a47675f82c66e66a24772a72cdb418816e72fa50c42c433f56"}
Oct 31 01:29:25.292838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.292704    5419 generic.go:356] "Generic (PLEG): container finished" podID="8bf46c21-01a3-4114-b531-cf64bfe7d7c2" containerID="05f8b201587c1cac1bbd92209be59843526b473404896173f0ef9a171b864d9c" exitCode=0
Oct 31 01:29:25.293644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.293499    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8" event={"ID":"8bf46c21-01a3-4114-b531-cf64bfe7d7c2","Type":"ContainerDied","Data":"05f8b201587c1cac1bbd92209be59843526b473404896173f0ef9a171b864d9c"}
Oct 31 01:29:25.791453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.791415    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"]
Oct 31 01:29:25.797203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.797174    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.844455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.844355    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"]
Oct 31 01:29:25.871179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.870059    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.871179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.870110    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlsc\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.871179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.870175    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.871179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.870208    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.871179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.870270    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.871179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.870354    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.885163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.879382    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.885163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.879486    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.885163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.879633    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:25.963161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.962420    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-14ca09-openshift-default-dockercfg-r7fdz\""
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995119    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995199    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995259    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995429    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995556    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995589    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlsc\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995677    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995719    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.000168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:25.995832    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.031164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.016750    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.031164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.017423    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.031164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.017829    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.061285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.031844    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.082300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.043065    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.143191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.138905    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.182182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.175853    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlsc\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.295163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.287847    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.347220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.344701    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token\") pod \"e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") " pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.498437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.496430    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:26.867403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.867378    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:26.923030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.922993    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw\" (UniqueName: \"kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw\") pod \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") "
Oct 31 01:29:26.923245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.923057    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca\") pod \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") "
Oct 31 01:29:26.923245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.923241    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca\") pod \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\" (UID: \"8bf46c21-01a3-4114-b531-cf64bfe7d7c2\") "
Oct 31 01:29:26.923640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.923595    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca" pod "8bf46c21-01a3-4114-b531-cf64bfe7d7c2" (UID: "8bf46c21-01a3-4114-b531-cf64bfe7d7c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:29:26.925433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.925402    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw" pod "8bf46c21-01a3-4114-b531-cf64bfe7d7c2" (UID: "8bf46c21-01a3-4114-b531-cf64bfe7d7c2"). InnerVolumeSpecName "kube-api-access-6z8hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:26.925757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:26.925743    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca" pod "8bf46c21-01a3-4114-b531-cf64bfe7d7c2" (UID: "8bf46c21-01a3-4114-b531-cf64bfe7d7c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:29:27.025426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.025400    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z8hw\" (UniqueName: \"kubernetes.io/projected/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-kube-api-access-6z8hw\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:27.025426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.025426    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:27.025633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.025436    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf46c21-01a3-4114-b531-cf64bfe7d7c2-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:29:27.152538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.152508    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"]
Oct 31 01:29:27.152795 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:29:27.152768    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5073b593_a911_413c_8c9f_22226ee3e0f7.slice/crio-cbc3bda2af37090ec21824a3cebb677178ac67b408b2043abe1ec3ec62c6cae0 WatchSource:0}: Error finding container cbc3bda2af37090ec21824a3cebb677178ac67b408b2043abe1ec3ec62c6cae0: Status 404 returned error can't find the container with id cbc3bda2af37090ec21824a3cebb677178ac67b408b2043abe1ec3ec62c6cae0
Oct 31 01:29:27.155676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.155643    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:27.155798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.155712    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:27.155798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.155740    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:27.421629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.421545    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" event={"ID":"5073b593-a911-413c-8c9f-22226ee3e0f7","Type":"ContainerStarted","Data":"752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"}
Oct 31 01:29:27.422045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.421648    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" event={"ID":"5073b593-a911-413c-8c9f-22226ee3e0f7","Type":"ContainerStarted","Data":"cbc3bda2af37090ec21824a3cebb677178ac67b408b2043abe1ec3ec62c6cae0"}
Oct 31 01:29:27.423197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.423170    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8" event={"ID":"8bf46c21-01a3-4114-b531-cf64bfe7d7c2","Type":"ContainerDied","Data":"a9feb05e923516a47675f82c66e66a24772a72cdb418816e72fa50c42c433f56"}
Oct 31 01:29:27.423327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.423216    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9feb05e923516a47675f82c66e66a24772a72cdb418816e72fa50c42c433f56"
Oct 31 01:29:27.423327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.423180    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"
Oct 31 01:29:27.498167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.497230    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:27.538343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:27.538290    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podStartSLOduration=2.538273055 podStartE2EDuration="2.538273055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:27.537163118 +0000 UTC m=+1488.838119689" watchObservedRunningTime="2026-10-31 01:29:27.538273055 +0000 UTC m=+1488.839229643"
Oct 31 01:29:28.511650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:28.511620    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:29.555649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.555617    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:29.561354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.561331    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:29:29.759163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.747618    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"]
Oct 31 01:29:29.784164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.760595    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8bf46c21-01a3-4114-b531-cf64bfe7d7c2" containerName="revoke-keys"
Oct 31 01:29:29.784422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.784391    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf46c21-01a3-4114-b531-cf64bfe7d7c2" containerName="revoke-keys"
Oct 31 01:29:29.794833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.785012    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf46c21-01a3-4114-b531-cf64bfe7d7c2" containerName="revoke-keys"
Oct 31 01:29:29.823687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.806431    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:29.877076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.877039    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c70e7-dockercfg-c5npv\""
Oct 31 01:29:29.909169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.898261    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c70e7\""
Oct 31 01:29:29.938097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.938007    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"]
Oct 31 01:29:29.952583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.952194    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:29.952583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:29.952358    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gllw6\" (UniqueName: \"kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6\") pod \"payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:30.055838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:30.055793    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:30.056974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:30.056848    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:30.057307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:30.057284    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gllw6\" (UniqueName: \"kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6\") pod \"payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:30.114355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:30.114287    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gllw6\" (UniqueName: \"kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6\") pod \"payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:30.202802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:30.202773    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:30.592230 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:29:30.592193    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f72ac5a_9e67_4e4b_8dff_743d048811f9.slice/crio-fa1f07a0187cf71996a49288f798f8308d5b0bdc1ffbacf7455e9112afa37652 WatchSource:0}: Error finding container fa1f07a0187cf71996a49288f798f8308d5b0bdc1ffbacf7455e9112afa37652: Status 404 returned error can't find the container with id fa1f07a0187cf71996a49288f798f8308d5b0bdc1ffbacf7455e9112afa37652
Oct 31 01:29:30.593270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:30.593244    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"]
Oct 31 01:29:31.096945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:31.095920    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"]
Oct 31 01:29:31.149909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:31.149840    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24111e67-7n28q"]
Oct 31 01:29:31.573666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:31.573629    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9" event={"ID":"7f72ac5a-9e67-4e4b-8dff-743d048811f9","Type":"ContainerStarted","Data":"f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920"}
Oct 31 01:29:31.573861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:31.573694    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9" event={"ID":"7f72ac5a-9e67-4e4b-8dff-743d048811f9","Type":"ContainerStarted","Data":"fa1f07a0187cf71996a49288f798f8308d5b0bdc1ffbacf7455e9112afa37652"}
Oct 31 01:29:31.691850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:31.691794    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9" podStartSLOduration=2.6917772920000003 podStartE2EDuration="2.691777292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:31.688280251 +0000 UTC m=+1492.989236827" watchObservedRunningTime="2026-10-31 01:29:31.691777292 +0000 UTC m=+1492.992733893"
Oct 31 01:29:32.047951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:32.047915    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"]
Oct 31 01:29:32.051292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:32.051262    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3567e9c-4lrbr"]
Oct 31 01:29:32.584939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:32.584908    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:32.843532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:32.843446    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccebd48e-7376-4c3a-9514-0d31fc7ba7c9" path="/var/lib/kubelet/pods/ccebd48e-7376-4c3a-9514-0d31fc7ba7c9/volumes"
Oct 31 01:29:32.844180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:32.844160    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f691c212-8599-498e-9b1b-a867345b7cc0" path="/var/lib/kubelet/pods/f691c212-8599-498e-9b1b-a867345b7cc0/volumes"
Oct 31 01:29:34.602401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:34.602370    5419 generic.go:356] "Generic (PLEG): container finished" podID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerID="6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b" exitCode=0
Oct 31 01:29:34.602849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:34.602444    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" event={"ID":"f0ab9da4-de4b-45a1-a08a-06f7def7d064","Type":"ContainerDied","Data":"6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"}
Oct 31 01:29:34.602906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:34.602865    5419 scope.go:122] "RemoveContainer" containerID="6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"
Oct 31 01:29:34.604453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:34.604428    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:34.604511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:34.604491    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:34.604542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:34.604525    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:29:35.612648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:35.612614    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" event={"ID":"f0ab9da4-de4b-45a1-a08a-06f7def7d064","Type":"ContainerStarted","Data":"c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0"}
Oct 31 01:29:35.710273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:35.710244    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:29:35.871926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:35.871853    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:29:35.873235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:35.873208    5419 patch_prober.go:29] interesting pod/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:29:35.873343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:35.873260    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 01:29:36.783165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:36.782847    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"]
Oct 31 01:29:37.205219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.001625    5419 patch_prober.go:29] interesting pod/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:29:37.205219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.001696    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 01:29:37.243207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.234869    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.363982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.363713    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"]
Oct 31 01:29:37.369533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.369508    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14ca09\""
Oct 31 01:29:37.369737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.369610    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14ca09-dockercfg-lhbcx\""
Oct 31 01:29:37.376420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.376387    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.376551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.376531    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8b2z\" (UniqueName: \"kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z\") pod \"payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.500274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.500208    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8b2z\" (UniqueName: \"kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z\") pod \"payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.500502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.500446    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.501445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.501417    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.557719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.557682    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8b2z\" (UniqueName: \"kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z\") pod \"payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.699449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.699395    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:37.871884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.871849    5419 patch_prober.go:29] interesting pod/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:29:37.872364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:37.871907    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 01:29:38.137297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.137267    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"]
Oct 31 01:29:38.797165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.796780    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5" event={"ID":"2d635118-4d0f-48ef-a9d3-e0f39752c250","Type":"ContainerStarted","Data":"b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793"}
Oct 31 01:29:38.797165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.796850    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5" event={"ID":"2d635118-4d0f-48ef-a9d3-e0f39752c250","Type":"ContainerStarted","Data":"7b506de4ad8500e44bbb9191fae4ddc3f5aac538b977c45bc95953621bffa8e6"}
Oct 31 01:29:38.800165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.798018    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:29:38.882387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.882350    5419 patch_prober.go:29] interesting pod/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:29:38.882985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.882955    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 01:29:38.966456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:38.965687    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5" podStartSLOduration=2.965663068 podStartE2EDuration="2.965663068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:38.94435064 +0000 UTC m=+1500.245307230" watchObservedRunningTime="2026-10-31 01:29:38.965663068 +0000 UTC m=+1500.266619645"
Oct 31 01:29:39.348295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:39.348259    5419 scope.go:122] "RemoveContainer" containerID="ec2c4e3e983da02bedfbea8c9d2607f0c665c1cb5628a4a7c135f7574c031bba"
Oct 31 01:29:39.359278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:39.359254    5419 scope.go:122] "RemoveContainer" containerID="74db30d4c5cdd8d4a7482a438786f3807b283a44609ebdf76dd534d76a57f52f"
Oct 31 01:29:39.369074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:39.369052    5419 scope.go:122] "RemoveContainer" containerID="b3779aa336a28536f622ba593784b34aab22c9111a4744cc738a42f6681ee7c6"
Oct 31 01:29:39.402213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:39.402185    5419 scope.go:122] "RemoveContainer" containerID="380ce2f8cab002df5cb982a90eb8cc0c1bcc92a878238faa526c2559bcffb92b"
Oct 31 01:29:39.872367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:39.872335    5419 patch_prober.go:29] interesting pod/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:29:39.872542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:39.872400    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 01:29:40.872849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:40.872811    5419 patch_prober.go:29] interesting pod/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Oct 31 01:29:40.873294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:40.872872    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Oct 31 01:29:41.057708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:41.057668    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"]
Oct 31 01:29:41.109799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:41.109757    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f027a4d-hj22g"]
Oct 31 01:29:41.876500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:41.876471    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:29:41.877493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:41.877467    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:29:42.838379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:42.838350    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3811269-2982-43fe-bde6-101768c44224" path="/var/lib/kubelet/pods/e3811269-2982-43fe-bde6-101768c44224/volumes"
Oct 31 01:29:43.050065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:43.050032    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"]
Oct 31 01:29:43.100196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:43.100104    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a6708a-bmwq7"]
Oct 31 01:29:43.103625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:43.103608    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"]
Oct 31 01:29:43.106499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:43.106479    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ebd6f9-856kl"]
Oct 31 01:29:43.597239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:43.597210    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:29:44.586122 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:29:44.586076    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:29:44.838789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:44.838713    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a9bf4a-232e-484c-b26a-f5601f2e81a0" path="/var/lib/kubelet/pods/43a9bf4a-232e-484c-b26a-f5601f2e81a0/volumes"
Oct 31 01:29:44.839167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:44.839152    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a6ed33-25a4-46bd-87d8-5e20797cd85a" path="/var/lib/kubelet/pods/e0a6ed33-25a4-46bd-87d8-5e20797cd85a/volumes"
Oct 31 01:29:50.818394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:29:50.818361    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:30:00.047261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.047230    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"]
Oct 31 01:30:00.095360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.095327    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8686b2b-lngbk"]
Oct 31 01:30:00.160502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.160465    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"]
Oct 31 01:30:00.165484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.165458    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"]
Oct 31 01:30:00.165672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.165577    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.169620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.169599    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"]
Oct 31 01:30:00.169756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.169734    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.173827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.173805    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"]
Oct 31 01:30:00.173944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.173859    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.178686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.178664    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"]
Oct 31 01:30:00.178813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.178774    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.183731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.183709    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"]
Oct 31 01:30:00.183847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.183818    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.188721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.188701    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"]
Oct 31 01:30:00.188815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.188789    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.193325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.193303    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.221423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.221398    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"]
Oct 31 01:30:00.222694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.222635    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:30:00.222871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.222810    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:30:00.222980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.222963    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:30:00.226804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.226781    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"]
Oct 31 01:30:00.281589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.281439    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"]
Oct 31 01:30:00.282794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.282720    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjc9\" (UniqueName: \"kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.283053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283030    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62pw\" (UniqueName: \"kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw\") pod \"maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.283205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283123    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.283205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283178    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.283376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283220    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.283376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283307    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.283376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283346    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5wlc\" (UniqueName: \"kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.283376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283378    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.283764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283412    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfj4\" (UniqueName: \"kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4\") pod \"maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.283764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283456    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqv4h\" (UniqueName: \"kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.283764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283509    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.283764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283612    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcdz\" (UniqueName: \"kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.283764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283713    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7rs\" (UniqueName: \"kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.284041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.283769    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.298495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.298413    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"]
Oct 31 01:30:00.304811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.304779    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"]
Oct 31 01:30:00.310400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.310372    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"]
Oct 31 01:30:00.339594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.339563    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"]
Oct 31 01:30:00.385415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385385    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62pw\" (UniqueName: \"kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw\") pod \"maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.385577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385458    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.385577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385488    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.385577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385512    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.385693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385586    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.385693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385613    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5wlc\" (UniqueName: \"kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.385693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385640    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.385788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385743    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfj4\" (UniqueName: \"kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4\") pod \"maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.385822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385794    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqv4h\" (UniqueName: \"kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.385857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385846    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.385972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.385943    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcdz\" (UniqueName: \"kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.386064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386051    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7rs\" (UniqueName: \"kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.386127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386098    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.386270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386179    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjc9\" (UniqueName: \"kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.386361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386329    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.386449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386423    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.386564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386424    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.386564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386458    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.386899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386855    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.386993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.386914    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.387216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.387199    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.443249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.443201    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62pw\" (UniqueName: \"kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw\") pod \"maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.443597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.443534    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqv4h\" (UniqueName: \"kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.443597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.443564    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcdz\" (UniqueName: \"kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.444370 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.444332    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5wlc\" (UniqueName: \"kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.444482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.444469    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7rs\" (UniqueName: \"kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.444595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.444579    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjc9\" (UniqueName: \"kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.444899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.444883    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfj4\" (UniqueName: \"kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4\") pod \"maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.483920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.483892    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:00.492346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.492328    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:00.502487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.502460    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:00.511387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.511358    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:00.523019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.522991    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:00.532301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.532272    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:00.540239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.540199    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:00.842478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.840924    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257c8c51-806f-4fca-a2ff-e68d0c064349" path="/var/lib/kubelet/pods/257c8c51-806f-4fca-a2ff-e68d0c064349/volumes"
Oct 31 01:30:00.886034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:00.885326    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"]
Oct 31 01:30:01.022818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.022787    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"]
Oct 31 01:30:01.041069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.041048    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"]
Oct 31 01:30:01.062689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.062654    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerStarted","Data":"c6384aa5b3f3a64ac8406d7331d81956d2f857207af03baa4aa3f74b920bba0d"}
Oct 31 01:30:01.076773 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:30:01.076729    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a1d8b5_8f22_4a71_aa95_61842e00abf8.slice/crio-36863c2a4c5feb67b8eca9516c7e06ae9519bf43dd756b138140e9c3e0c84a0d WatchSource:0}: Error finding container 36863c2a4c5feb67b8eca9516c7e06ae9519bf43dd756b138140e9c3e0c84a0d: Status 404 returned error can't find the container with id 36863c2a4c5feb67b8eca9516c7e06ae9519bf43dd756b138140e9c3e0c84a0d
Oct 31 01:30:01.077559 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:30:01.077535    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e2f664_bc53_437f_833a_982c2b7f1a21.slice/crio-92ae09906f74090f9ca5ea132b4088ae31b05c5876fb3f0d5a369e6e8ebe8641 WatchSource:0}: Error finding container 92ae09906f74090f9ca5ea132b4088ae31b05c5876fb3f0d5a369e6e8ebe8641: Status 404 returned error can't find the container with id 92ae09906f74090f9ca5ea132b4088ae31b05c5876fb3f0d5a369e6e8ebe8641
Oct 31 01:30:01.128116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.128082    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"]
Oct 31 01:30:01.128287 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:30:01.128263    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906ed670_89e6_42b5_a18d_ec1acad34231.slice/crio-b75e08473052ae2d3e90d05b90ee9b9abd03ddc8ef09a9eead17894ee475efa0 WatchSource:0}: Error finding container b75e08473052ae2d3e90d05b90ee9b9abd03ddc8ef09a9eead17894ee475efa0: Status 404 returned error can't find the container with id b75e08473052ae2d3e90d05b90ee9b9abd03ddc8ef09a9eead17894ee475efa0
Oct 31 01:30:01.129820 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:30:01.129798    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2042ee_34cc_413a_bcf5_ff04ae176627.slice/crio-f767fbdc625f583fd90298820f27fdcc0a02bcc82aeb7d23e552da07ef53a2ab WatchSource:0}: Error finding container f767fbdc625f583fd90298820f27fdcc0a02bcc82aeb7d23e552da07ef53a2ab: Status 404 returned error can't find the container with id f767fbdc625f583fd90298820f27fdcc0a02bcc82aeb7d23e552da07ef53a2ab
Oct 31 01:30:01.132091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.132070    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"]
Oct 31 01:30:01.165183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.165129    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"]
Oct 31 01:30:01.165749 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:30:01.165727    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa841e8_26f3_4038_a2a8_b1c832ba35f3.slice/crio-0b953fe07a7d18788e1433e47ae8bd3609c94b9427b25809d7eef0591cabdf73 WatchSource:0}: Error finding container 0b953fe07a7d18788e1433e47ae8bd3609c94b9427b25809d7eef0591cabdf73: Status 404 returned error can't find the container with id 0b953fe07a7d18788e1433e47ae8bd3609c94b9427b25809d7eef0591cabdf73
Oct 31 01:30:01.173432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:01.173408    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"]
Oct 31 01:30:01.177094 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:30:01.177066    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a960de_8acf_4679_85e2_b560a9a2c1e8.slice/crio-ab598776e3a33273a2cbcd85be7ab1cc80cab648b11fe6091d0ed0e4cce47efb WatchSource:0}: Error finding container ab598776e3a33273a2cbcd85be7ab1cc80cab648b11fe6091d0ed0e4cce47efb: Status 404 returned error can't find the container with id ab598776e3a33273a2cbcd85be7ab1cc80cab648b11fe6091d0ed0e4cce47efb
Oct 31 01:30:02.072088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.072054    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerStarted","Data":"b5ed2011bf737031a59270a91071fc5a7574bd0e00a8af364dce939ac14067e0"}
Oct 31 01:30:02.072088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.072104    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerStarted","Data":"f767fbdc625f583fd90298820f27fdcc0a02bcc82aeb7d23e552da07ef53a2ab"}
Oct 31 01:30:02.073891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.073860    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerStarted","Data":"4e730f932297fdeea9030ce3f76482beae89262b4c30016756e102a8acabf664"}
Oct 31 01:30:02.074035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.073911    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerStarted","Data":"0b953fe07a7d18788e1433e47ae8bd3609c94b9427b25809d7eef0591cabdf73"}
Oct 31 01:30:02.078576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.078551    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerStarted","Data":"b494664a75d5c9cad25062b79f4a25743dbc8fa73d9382b78db66a2addca96cf"}
Oct 31 01:30:02.078711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.078603    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerStarted","Data":"92ae09906f74090f9ca5ea132b4088ae31b05c5876fb3f0d5a369e6e8ebe8641"}
Oct 31 01:30:02.081346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.081312    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerStarted","Data":"35da9da2b60a2f763471a5d5fd16b632e66d4419f7f2e5be19664fa6ad47c172"}
Oct 31 01:30:02.081460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.081373    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerStarted","Data":"ab598776e3a33273a2cbcd85be7ab1cc80cab648b11fe6091d0ed0e4cce47efb"}
Oct 31 01:30:02.083331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.083307    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerStarted","Data":"e9cb3a51b56bcbb4aa36ce778eef5a732aece048b12566a8a73c6735bcd5e907"}
Oct 31 01:30:02.083440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.083350    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerStarted","Data":"b75e08473052ae2d3e90d05b90ee9b9abd03ddc8ef09a9eead17894ee475efa0"}
Oct 31 01:30:02.085212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.085130    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerStarted","Data":"f6c6e6746553ef76054425f77c2c151b19a8212f368ad4168be99a32f078543a"}
Oct 31 01:30:02.086804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.086783    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" event={"ID":"b9a1d8b5-8f22-4a71-aa95-61842e00abf8","Type":"ContainerStarted","Data":"3c66039258fc18356cf623ac800bea8999d96d6ce49c5a1f54a2535ed03bfc83"}
Oct 31 01:30:02.086888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.086830    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" event={"ID":"b9a1d8b5-8f22-4a71-aa95-61842e00abf8","Type":"ContainerStarted","Data":"36863c2a4c5feb67b8eca9516c7e06ae9519bf43dd756b138140e9c3e0c84a0d"}
Oct 31 01:30:02.185601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.185546    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" podStartSLOduration=2.185533375 podStartE2EDuration="2.185533375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.184400699 +0000 UTC m=+1523.485357276" watchObservedRunningTime="2026-10-31 01:30:02.185533375 +0000 UTC m=+1523.486489951"
Oct 31 01:30:02.297994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.297946    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" podStartSLOduration=2.297932319 podStartE2EDuration="2.297932319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.29710422 +0000 UTC m=+1523.598060811" watchObservedRunningTime="2026-10-31 01:30:02.297932319 +0000 UTC m=+1523.598888895"
Oct 31 01:30:02.411969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.411911    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" podStartSLOduration=2.411894897 podStartE2EDuration="2.411894897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.40984523 +0000 UTC m=+1523.710801818" watchObservedRunningTime="2026-10-31 01:30:02.411894897 +0000 UTC m=+1523.712851478"
Oct 31 01:30:02.529444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.529393    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" podStartSLOduration=2.529378238 podStartE2EDuration="2.529378238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.525571014 +0000 UTC m=+1523.826527594" watchObservedRunningTime="2026-10-31 01:30:02.529378238 +0000 UTC m=+1523.830334815"
Oct 31 01:30:02.638514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.638416    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" podStartSLOduration=2.638402223 podStartE2EDuration="2.638402223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.636671901 +0000 UTC m=+1523.937628477" watchObservedRunningTime="2026-10-31 01:30:02.638402223 +0000 UTC m=+1523.939358800"
Oct 31 01:30:02.750219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.750169    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" podStartSLOduration=2.750132985 podStartE2EDuration="2.750132985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.748481629 +0000 UTC m=+1524.049438208" watchObservedRunningTime="2026-10-31 01:30:02.750132985 +0000 UTC m=+1524.051089559"
Oct 31 01:30:02.866949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:02.866899    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" podStartSLOduration=2.866885661 podStartE2EDuration="2.866885661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.863729762 +0000 UTC m=+1524.164686342" watchObservedRunningTime="2026-10-31 01:30:02.866885661 +0000 UTC m=+1524.167842239"
Oct 31 01:30:03.096415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.096382    5419 generic.go:356] "Generic (PLEG): container finished" podID="906ed670-89e6-42b5-a18d-ec1acad34231" containerID="e9cb3a51b56bcbb4aa36ce778eef5a732aece048b12566a8a73c6735bcd5e907" exitCode=60
Oct 31 01:30:03.096848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.096461    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerDied","Data":"e9cb3a51b56bcbb4aa36ce778eef5a732aece048b12566a8a73c6735bcd5e907"}
Oct 31 01:30:03.096848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.096800    5419 scope.go:122] "RemoveContainer" containerID="e9cb3a51b56bcbb4aa36ce778eef5a732aece048b12566a8a73c6735bcd5e907"
Oct 31 01:30:03.100247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.100213    5419 generic.go:356] "Generic (PLEG): container finished" podID="77b67658-bc01-48df-b1cf-beef150b68ef" containerID="f6c6e6746553ef76054425f77c2c151b19a8212f368ad4168be99a32f078543a" exitCode=60
Oct 31 01:30:03.100358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.100294    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerDied","Data":"f6c6e6746553ef76054425f77c2c151b19a8212f368ad4168be99a32f078543a"}
Oct 31 01:30:03.100627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.100610    5419 scope.go:122] "RemoveContainer" containerID="f6c6e6746553ef76054425f77c2c151b19a8212f368ad4168be99a32f078543a"
Oct 31 01:30:03.102101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.102076    5419 generic.go:356] "Generic (PLEG): container finished" podID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerID="3c66039258fc18356cf623ac800bea8999d96d6ce49c5a1f54a2535ed03bfc83" exitCode=60
Oct 31 01:30:03.102240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.102105    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" event={"ID":"b9a1d8b5-8f22-4a71-aa95-61842e00abf8","Type":"ContainerDied","Data":"3c66039258fc18356cf623ac800bea8999d96d6ce49c5a1f54a2535ed03bfc83"}
Oct 31 01:30:03.102433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.102417    5419 scope.go:122] "RemoveContainer" containerID="3c66039258fc18356cf623ac800bea8999d96d6ce49c5a1f54a2535ed03bfc83"
Oct 31 01:30:03.103829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.103810    5419 generic.go:356] "Generic (PLEG): container finished" podID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerID="b5ed2011bf737031a59270a91071fc5a7574bd0e00a8af364dce939ac14067e0" exitCode=60
Oct 31 01:30:03.103930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.103891    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerDied","Data":"b5ed2011bf737031a59270a91071fc5a7574bd0e00a8af364dce939ac14067e0"}
Oct 31 01:30:03.104103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.104088    5419 scope.go:122] "RemoveContainer" containerID="b5ed2011bf737031a59270a91071fc5a7574bd0e00a8af364dce939ac14067e0"
Oct 31 01:30:03.105607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.105582    5419 generic.go:356] "Generic (PLEG): container finished" podID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerID="4e730f932297fdeea9030ce3f76482beae89262b4c30016756e102a8acabf664" exitCode=60
Oct 31 01:30:03.105705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.105671    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerDied","Data":"4e730f932297fdeea9030ce3f76482beae89262b4c30016756e102a8acabf664"}
Oct 31 01:30:03.105861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.105842    5419 scope.go:122] "RemoveContainer" containerID="4e730f932297fdeea9030ce3f76482beae89262b4c30016756e102a8acabf664"
Oct 31 01:30:03.107386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.107369    5419 generic.go:356] "Generic (PLEG): container finished" podID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerID="b494664a75d5c9cad25062b79f4a25743dbc8fa73d9382b78db66a2addca96cf" exitCode=60
Oct 31 01:30:03.107492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.107404    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerDied","Data":"b494664a75d5c9cad25062b79f4a25743dbc8fa73d9382b78db66a2addca96cf"}
Oct 31 01:30:03.107638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.107623    5419 scope.go:122] "RemoveContainer" containerID="b494664a75d5c9cad25062b79f4a25743dbc8fa73d9382b78db66a2addca96cf"
Oct 31 01:30:03.110670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.110645    5419 generic.go:356] "Generic (PLEG): container finished" podID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerID="35da9da2b60a2f763471a5d5fd16b632e66d4419f7f2e5be19664fa6ad47c172" exitCode=60
Oct 31 01:30:03.110770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.110686    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerDied","Data":"35da9da2b60a2f763471a5d5fd16b632e66d4419f7f2e5be19664fa6ad47c172"}
Oct 31 01:30:03.111101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:03.110999    5419 scope.go:122] "RemoveContainer" containerID="35da9da2b60a2f763471a5d5fd16b632e66d4419f7f2e5be19664fa6ad47c172"
Oct 31 01:30:04.120371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.120289    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerStarted","Data":"d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9"}
Oct 31 01:30:04.122479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.122457    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerStarted","Data":"a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614"}
Oct 31 01:30:04.124470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.124446    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" event={"ID":"b9a1d8b5-8f22-4a71-aa95-61842e00abf8","Type":"ContainerStarted","Data":"97ac01337438775f9a50a7e6272b88f816d10f324337a9c981709492ff228662"}
Oct 31 01:30:04.126338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.126318    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerStarted","Data":"009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec"}
Oct 31 01:30:04.128334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.128315    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerStarted","Data":"ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51"}
Oct 31 01:30:04.130121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.130099    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerStarted","Data":"78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556"}
Oct 31 01:30:04.131723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.131702    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerStarted","Data":"d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0"}
Oct 31 01:30:04.632854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:04.632794    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:30:05.141662 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.141627    5419 generic.go:356] "Generic (PLEG): container finished" podID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerID="009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec" exitCode=60
Oct 31 01:30:05.142080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.141708    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerDied","Data":"009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec"}
Oct 31 01:30:05.142080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.141763    5419 scope.go:122] "RemoveContainer" containerID="b5ed2011bf737031a59270a91071fc5a7574bd0e00a8af364dce939ac14067e0"
Oct 31 01:30:05.142080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.142044    5419 scope.go:122] "RemoveContainer" containerID="009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec"
Oct 31 01:30:05.142467 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.142441    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf_odh-ai-gateway-infra(fd2042ee-34cc-413a-bcf5-ff04ae176627)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627"
Oct 31 01:30:05.143907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.143884    5419 generic.go:356] "Generic (PLEG): container finished" podID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerID="ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51" exitCode=60
Oct 31 01:30:05.144001 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.143961    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerDied","Data":"ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51"}
Oct 31 01:30:05.144240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.144222    5419 scope.go:122] "RemoveContainer" containerID="ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51"
Oct 31 01:30:05.144538 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.144516    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp_odh-ai-gateway-infra(dfa841e8-26f3-4038-a2a8-b1c832ba35f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3"
Oct 31 01:30:05.145735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.145713    5419 generic.go:356] "Generic (PLEG): container finished" podID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerID="78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556" exitCode=60
Oct 31 01:30:05.145842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.145774    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerDied","Data":"78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556"}
Oct 31 01:30:05.146020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.146006    5419 scope.go:122] "RemoveContainer" containerID="78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556"
Oct 31 01:30:05.146345 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.146320    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb_odh-ai-gateway-infra(68e2f664-bc53-437f-833a-982c2b7f1a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21"
Oct 31 01:30:05.147776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.147754    5419 generic.go:356] "Generic (PLEG): container finished" podID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerID="d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0" exitCode=60
Oct 31 01:30:05.147854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.147788    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerDied","Data":"d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0"}
Oct 31 01:30:05.148056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.148040    5419 scope.go:122] "RemoveContainer" containerID="d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0"
Oct 31 01:30:05.148389 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.148369    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849_odh-ai-gateway-infra(80a960de-8acf-4679-85e2-b560a9a2c1e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8"
Oct 31 01:30:05.149902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.149881    5419 generic.go:356] "Generic (PLEG): container finished" podID="906ed670-89e6-42b5-a18d-ec1acad34231" containerID="d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9" exitCode=60
Oct 31 01:30:05.150001 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.149963    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerDied","Data":"d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9"}
Oct 31 01:30:05.150227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.150206    5419 scope.go:122] "RemoveContainer" containerID="d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9"
Oct 31 01:30:05.150478 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.150459    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5_odh-ai-gateway-infra(906ed670-89e6-42b5-a18d-ec1acad34231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" podUID="906ed670-89e6-42b5-a18d-ec1acad34231"
Oct 31 01:30:05.152033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.152010    5419 generic.go:356] "Generic (PLEG): container finished" podID="77b67658-bc01-48df-b1cf-beef150b68ef" containerID="a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614" exitCode=60
Oct 31 01:30:05.152165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.152038    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerDied","Data":"a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614"}
Oct 31 01:30:05.152539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.152524    5419 scope.go:122] "RemoveContainer" containerID="a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614"
Oct 31 01:30:05.152778 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.152762    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln_odh-ai-gateway-infra(77b67658-bc01-48df-b1cf-beef150b68ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" podUID="77b67658-bc01-48df-b1cf-beef150b68ef"
Oct 31 01:30:05.154008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.153989    5419 generic.go:356] "Generic (PLEG): container finished" podID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerID="97ac01337438775f9a50a7e6272b88f816d10f324337a9c981709492ff228662" exitCode=60
Oct 31 01:30:05.154104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.154069    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" event={"ID":"b9a1d8b5-8f22-4a71-aa95-61842e00abf8","Type":"ContainerDied","Data":"97ac01337438775f9a50a7e6272b88f816d10f324337a9c981709492ff228662"}
Oct 31 01:30:05.154269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.154256    5419 scope.go:122] "RemoveContainer" containerID="97ac01337438775f9a50a7e6272b88f816d10f324337a9c981709492ff228662"
Oct 31 01:30:05.154465 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:05.154449    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm_odh-ai-gateway-infra(b9a1d8b5-8f22-4a71-aa95-61842e00abf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8"
Oct 31 01:30:05.163966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.163943    5419 scope.go:122] "RemoveContainer" containerID="4e730f932297fdeea9030ce3f76482beae89262b4c30016756e102a8acabf664"
Oct 31 01:30:05.177850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.177830    5419 scope.go:122] "RemoveContainer" containerID="b494664a75d5c9cad25062b79f4a25743dbc8fa73d9382b78db66a2addca96cf"
Oct 31 01:30:05.191202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.191184    5419 scope.go:122] "RemoveContainer" containerID="35da9da2b60a2f763471a5d5fd16b632e66d4419f7f2e5be19664fa6ad47c172"
Oct 31 01:30:05.204628 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.204595    5419 scope.go:122] "RemoveContainer" containerID="e9cb3a51b56bcbb4aa36ce778eef5a732aece048b12566a8a73c6735bcd5e907"
Oct 31 01:30:05.218199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.218175    5419 scope.go:122] "RemoveContainer" containerID="f6c6e6746553ef76054425f77c2c151b19a8212f368ad4168be99a32f078543a"
Oct 31 01:30:05.231392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:05.231370    5419 scope.go:122] "RemoveContainer" containerID="3c66039258fc18356cf623ac800bea8999d96d6ce49c5a1f54a2535ed03bfc83"
Oct 31 01:30:06.163980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.163950    5419 scope.go:122] "RemoveContainer" containerID="d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0"
Oct 31 01:30:06.164448 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.164329    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849_odh-ai-gateway-infra(80a960de-8acf-4679-85e2-b560a9a2c1e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8"
Oct 31 01:30:06.165564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.165546    5419 scope.go:122] "RemoveContainer" containerID="d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9"
Oct 31 01:30:06.165799 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.165781    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5_odh-ai-gateway-infra(906ed670-89e6-42b5-a18d-ec1acad34231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" podUID="906ed670-89e6-42b5-a18d-ec1acad34231"
Oct 31 01:30:06.167104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.167087    5419 scope.go:122] "RemoveContainer" containerID="a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614"
Oct 31 01:30:06.167338 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.167316    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln_odh-ai-gateway-infra(77b67658-bc01-48df-b1cf-beef150b68ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" podUID="77b67658-bc01-48df-b1cf-beef150b68ef"
Oct 31 01:30:06.168606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.168592    5419 scope.go:122] "RemoveContainer" containerID="97ac01337438775f9a50a7e6272b88f816d10f324337a9c981709492ff228662"
Oct 31 01:30:06.168813 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.168792    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm_odh-ai-gateway-infra(b9a1d8b5-8f22-4a71-aa95-61842e00abf8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8"
Oct 31 01:30:06.170273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.170249    5419 scope.go:122] "RemoveContainer" containerID="009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec"
Oct 31 01:30:06.170469 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.170449    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf_odh-ai-gateway-infra(fd2042ee-34cc-413a-bcf5-ff04ae176627)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627"
Oct 31 01:30:06.171882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.171863    5419 scope.go:122] "RemoveContainer" containerID="ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51"
Oct 31 01:30:06.172106 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.172077    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp_odh-ai-gateway-infra(dfa841e8-26f3-4038-a2a8-b1c832ba35f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3"
Oct 31 01:30:06.173451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:06.173436    5419 scope.go:122] "RemoveContainer" containerID="78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556"
Oct 31 01:30:06.173629 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:06.173612    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb_odh-ai-gateway-infra(68e2f664-bc53-437f-833a-982c2b7f1a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21"
Oct 31 01:30:11.059954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:11.059914    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"]
Oct 31 01:30:11.109016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:11.108984    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f597032c-4wzgf"]
Oct 31 01:30:12.839657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:12.839624    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632e3055-6be7-49dd-9edb-b16e13d17d72" path="/var/lib/kubelet/pods/632e3055-6be7-49dd-9edb-b16e13d17d72/volumes"
Oct 31 01:30:12.874822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:12.874784    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"]
Oct 31 01:30:13.417481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.417449    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:13.562460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.562372    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca\") pod \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") "
Oct 31 01:30:13.562629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.562465    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4\" (UniqueName: \"kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4\") pod \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\" (UID: \"b9a1d8b5-8f22-4a71-aa95-61842e00abf8\") "
Oct 31 01:30:13.562726 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.562702    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca" pod "b9a1d8b5-8f22-4a71-aa95-61842e00abf8" (UID: "b9a1d8b5-8f22-4a71-aa95-61842e00abf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:13.563245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.563224    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:13.564819 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.564787    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4" pod "b9a1d8b5-8f22-4a71-aa95-61842e00abf8" (UID: "b9a1d8b5-8f22-4a71-aa95-61842e00abf8"). InnerVolumeSpecName "kube-api-access-lpfj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:13.678162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:13.672223    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpfj4\" (UniqueName: \"kubernetes.io/projected/b9a1d8b5-8f22-4a71-aa95-61842e00abf8-kube-api-access-lpfj4\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:14.289389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.289356    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"
Oct 31 01:30:14.289389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.289384    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm" event={"ID":"b9a1d8b5-8f22-4a71-aa95-61842e00abf8","Type":"ContainerDied","Data":"36863c2a4c5feb67b8eca9516c7e06ae9519bf43dd756b138140e9c3e0c84a0d"}
Oct 31 01:30:14.289935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.289443    5419 scope.go:122] "RemoveContainer" containerID="97ac01337438775f9a50a7e6272b88f816d10f324337a9c981709492ff228662"
Oct 31 01:30:14.463679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.463650    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"]
Oct 31 01:30:14.512738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.512707    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-18bf762c-29890170-hkzvm"]
Oct 31 01:30:14.584070 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:14.583992    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:30:14.830542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.830515    5419 scope.go:122] "RemoveContainer" containerID="ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51"
Oct 31 01:30:14.843922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.843424    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" path="/var/lib/kubelet/pods/b9a1d8b5-8f22-4a71-aa95-61842e00abf8/volumes"
Oct 31 01:30:14.843922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.843868    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"]
Oct 31 01:30:14.847423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:14.847400    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18bf762c-9n5n4"]
Oct 31 01:30:15.300427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:15.300399    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerStarted","Data":"128e33b8cdb0b21c3dd06b0907ff962622fadddf405bfced4352b306d5971179"}
Oct 31 01:30:16.310872 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.310844    5419 generic.go:356] "Generic (PLEG): container finished" podID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerID="128e33b8cdb0b21c3dd06b0907ff962622fadddf405bfced4352b306d5971179" exitCode=60
Oct 31 01:30:16.311328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.310890    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerDied","Data":"128e33b8cdb0b21c3dd06b0907ff962622fadddf405bfced4352b306d5971179"}
Oct 31 01:30:16.311328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.310937    5419 scope.go:122] "RemoveContainer" containerID="ac45920800ccd03368a9c9623aca814437afb7f193821f04add2e74ddd9b3c51"
Oct 31 01:30:16.311429 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.311393    5419 scope.go:122] "RemoveContainer" containerID="128e33b8cdb0b21c3dd06b0907ff962622fadddf405bfced4352b306d5971179"
Oct 31 01:30:16.311802 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:16.311773    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp_odh-ai-gateway-infra(dfa841e8-26f3-4038-a2a8-b1c832ba35f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3"
Oct 31 01:30:16.427618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.427586    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"]
Oct 31 01:30:16.830416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.830390    5419 scope.go:122] "RemoveContainer" containerID="a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614"
Oct 31 01:30:16.830564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.830513    5419 scope.go:122] "RemoveContainer" containerID="78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556"
Oct 31 01:30:16.839068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:16.839042    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e693c0e-ea2d-4860-b2bb-6de8308d0c82" path="/var/lib/kubelet/pods/2e693c0e-ea2d-4860-b2bb-6de8308d0c82/volumes"
Oct 31 01:30:17.320988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.320957    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerStarted","Data":"65fc8c9b14c51fdf32634841d12000878cadb0eb0200740ba8d4c85c24a9afb4"}
Oct 31 01:30:17.324864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.324835    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerStarted","Data":"e76c85f50ec928261dbba6e95bed32be6b8f563fb3e080a1a43eab8f994e0fc8"}
Oct 31 01:30:17.505032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.505004    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:17.638331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.638246    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9\" (UniqueName: \"kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9\") pod \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") "
Oct 31 01:30:17.638624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.638604    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca\") pod \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\" (UID: \"dfa841e8-26f3-4038-a2a8-b1c832ba35f3\") "
Oct 31 01:30:17.639016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.638915    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca" pod "dfa841e8-26f3-4038-a2a8-b1c832ba35f3" (UID: "dfa841e8-26f3-4038-a2a8-b1c832ba35f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:17.639249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.639124    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:17.640663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.640636    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9" pod "dfa841e8-26f3-4038-a2a8-b1c832ba35f3" (UID: "dfa841e8-26f3-4038-a2a8-b1c832ba35f3"). InnerVolumeSpecName "kube-api-access-kvjc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:17.740753 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.740722    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvjc9\" (UniqueName: \"kubernetes.io/projected/dfa841e8-26f3-4038-a2a8-b1c832ba35f3-kube-api-access-kvjc9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:17.830757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.830730    5419 scope.go:122] "RemoveContainer" containerID="d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0"
Oct 31 01:30:17.830929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:17.830911    5419 scope.go:122] "RemoveContainer" containerID="009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec"
Oct 31 01:30:18.341408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.341370    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerStarted","Data":"c97306cfba6f1ec00afa55f2d6dab8b0ece7e4d763aac3855955303f764c29f9"}
Oct 31 01:30:18.343424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.343394    5419 generic.go:356] "Generic (PLEG): container finished" podID="77b67658-bc01-48df-b1cf-beef150b68ef" containerID="65fc8c9b14c51fdf32634841d12000878cadb0eb0200740ba8d4c85c24a9afb4" exitCode=60
Oct 31 01:30:18.343544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.343456    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerDied","Data":"65fc8c9b14c51fdf32634841d12000878cadb0eb0200740ba8d4c85c24a9afb4"}
Oct 31 01:30:18.343544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.343507    5419 scope.go:122] "RemoveContainer" containerID="a3180f8deb5c3bc7f1c45a6ec2f670fa1b926f5710c6e73eeda16e1d30a5f614"
Oct 31 01:30:18.343942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.343921    5419 scope.go:122] "RemoveContainer" containerID="65fc8c9b14c51fdf32634841d12000878cadb0eb0200740ba8d4c85c24a9afb4"
Oct 31 01:30:18.344384 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:18.344351    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln_odh-ai-gateway-infra(77b67658-bc01-48df-b1cf-beef150b68ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" podUID="77b67658-bc01-48df-b1cf-beef150b68ef"
Oct 31 01:30:18.345689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.345667    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerStarted","Data":"184d3147fddaed2144ea4562553c6f6d478774266859e541c660f944b25e779c"}
Oct 31 01:30:18.347103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.347055    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"
Oct 31 01:30:18.347196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.347064    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp" event={"ID":"dfa841e8-26f3-4038-a2a8-b1c832ba35f3","Type":"ContainerDied","Data":"0b953fe07a7d18788e1433e47ae8bd3609c94b9427b25809d7eef0591cabdf73"}
Oct 31 01:30:18.349217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.349196    5419 generic.go:356] "Generic (PLEG): container finished" podID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerID="e76c85f50ec928261dbba6e95bed32be6b8f563fb3e080a1a43eab8f994e0fc8" exitCode=60
Oct 31 01:30:18.349323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.349224    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerDied","Data":"e76c85f50ec928261dbba6e95bed32be6b8f563fb3e080a1a43eab8f994e0fc8"}
Oct 31 01:30:18.349555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.349538    5419 scope.go:122] "RemoveContainer" containerID="e76c85f50ec928261dbba6e95bed32be6b8f563fb3e080a1a43eab8f994e0fc8"
Oct 31 01:30:18.349817 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:18.349791    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb_odh-ai-gateway-infra(68e2f664-bc53-437f-833a-982c2b7f1a21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21"
Oct 31 01:30:18.359072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.359058    5419 scope.go:122] "RemoveContainer" containerID="128e33b8cdb0b21c3dd06b0907ff962622fadddf405bfced4352b306d5971179"
Oct 31 01:30:18.369939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.369922    5419 scope.go:122] "RemoveContainer" containerID="78c287c74129feb4c289866e1694c608b536e9296f8fa79d7cd7cc848a1b6556"
Oct 31 01:30:18.447719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.447686    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"]
Oct 31 01:30:18.560040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.559953    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"]
Oct 31 01:30:18.830856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.830785    5419 scope.go:122] "RemoveContainer" containerID="d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9"
Oct 31 01:30:18.912288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.912258    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"]
Oct 31 01:30:18.921581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:18.921557    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890170-hhtbp"]
Oct 31 01:30:19.360304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.360220    5419 generic.go:356] "Generic (PLEG): container finished" podID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerID="184d3147fddaed2144ea4562553c6f6d478774266859e541c660f944b25e779c" exitCode=60
Oct 31 01:30:19.360701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.360299    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerDied","Data":"184d3147fddaed2144ea4562553c6f6d478774266859e541c660f944b25e779c"}
Oct 31 01:30:19.360701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.360354    5419 scope.go:122] "RemoveContainer" containerID="009ebef10675c6d304c944b8220d766ae426d9953157ce3862d257a0481270ec"
Oct 31 01:30:19.360844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.360787    5419 scope.go:122] "RemoveContainer" containerID="184d3147fddaed2144ea4562553c6f6d478774266859e541c660f944b25e779c"
Oct 31 01:30:19.361238 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:19.361205    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf_odh-ai-gateway-infra(fd2042ee-34cc-413a-bcf5-ff04ae176627)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627"
Oct 31 01:30:19.365252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.365222    5419 generic.go:356] "Generic (PLEG): container finished" podID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerID="c97306cfba6f1ec00afa55f2d6dab8b0ece7e4d763aac3855955303f764c29f9" exitCode=60
Oct 31 01:30:19.365364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.365250    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerDied","Data":"c97306cfba6f1ec00afa55f2d6dab8b0ece7e4d763aac3855955303f764c29f9"}
Oct 31 01:30:19.365593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.365576    5419 scope.go:122] "RemoveContainer" containerID="c97306cfba6f1ec00afa55f2d6dab8b0ece7e4d763aac3855955303f764c29f9"
Oct 31 01:30:19.365896 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:19.365872    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849_odh-ai-gateway-infra(80a960de-8acf-4679-85e2-b560a9a2c1e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8"
Oct 31 01:30:19.367523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.367503    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerStarted","Data":"a5e6af385dc8cff9edd3a5d7901661ca6f3a9888c03d18efa241eae40e90f902"}
Oct 31 01:30:19.377014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.376996    5419 scope.go:122] "RemoveContainer" containerID="d8bd73ad51f3ba6d5b9f2eccc457db1155a7a6a7de12a1c8d32663f0c6208de0"
Oct 31 01:30:19.477477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.477439    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"]
Oct 31 01:30:19.559943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.559915    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:19.566358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.566325    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca\") pod \"68e2f664-bc53-437f-833a-982c2b7f1a21\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") "
Oct 31 01:30:19.566599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.566582    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h\" (UniqueName: \"kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h\") pod \"68e2f664-bc53-437f-833a-982c2b7f1a21\" (UID: \"68e2f664-bc53-437f-833a-982c2b7f1a21\") "
Oct 31 01:30:19.568569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.567997    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca" pod "68e2f664-bc53-437f-833a-982c2b7f1a21" (UID: "68e2f664-bc53-437f-833a-982c2b7f1a21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:19.583622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.583597    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h" pod "68e2f664-bc53-437f-833a-982c2b7f1a21" (UID: "68e2f664-bc53-437f-833a-982c2b7f1a21"). InnerVolumeSpecName "kube-api-access-mqv4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:19.613774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.613723    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:19.697165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.694540    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e2f664-bc53-437f-833a-982c2b7f1a21-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:19.697165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.694581    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqv4h\" (UniqueName: \"kubernetes.io/projected/68e2f664-bc53-437f-833a-982c2b7f1a21-kube-api-access-mqv4h\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:19.799969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.796835    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca\") pod \"77b67658-bc01-48df-b1cf-beef150b68ef\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") "
Oct 31 01:30:19.799969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.797229    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz\" (UniqueName: \"kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz\") pod \"77b67658-bc01-48df-b1cf-beef150b68ef\" (UID: \"77b67658-bc01-48df-b1cf-beef150b68ef\") "
Oct 31 01:30:19.799969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.799338    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca" pod "77b67658-bc01-48df-b1cf-beef150b68ef" (UID: "77b67658-bc01-48df-b1cf-beef150b68ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:19.801818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.801688    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz" pod "77b67658-bc01-48df-b1cf-beef150b68ef" (UID: "77b67658-bc01-48df-b1cf-beef150b68ef"). InnerVolumeSpecName "kube-api-access-vfcdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:19.834164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.812992    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"]
Oct 31 01:30:19.904222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.903557    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcdz\" (UniqueName: \"kubernetes.io/projected/77b67658-bc01-48df-b1cf-beef150b68ef-kube-api-access-vfcdz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:19.904222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:19.903622    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77b67658-bc01-48df-b1cf-beef150b68ef-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:20.232777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.232748    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"]
Oct 31 01:30:20.380271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.380244    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"
Oct 31 01:30:20.380651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.380283    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb" event={"ID":"68e2f664-bc53-437f-833a-982c2b7f1a21","Type":"ContainerDied","Data":"92ae09906f74090f9ca5ea132b4088ae31b05c5876fb3f0d5a369e6e8ebe8641"}
Oct 31 01:30:20.380651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.380345    5419 scope.go:122] "RemoveContainer" containerID="e76c85f50ec928261dbba6e95bed32be6b8f563fb3e080a1a43eab8f994e0fc8"
Oct 31 01:30:20.393662 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.393625    5419 generic.go:356] "Generic (PLEG): container finished" podID="906ed670-89e6-42b5-a18d-ec1acad34231" containerID="a5e6af385dc8cff9edd3a5d7901661ca6f3a9888c03d18efa241eae40e90f902" exitCode=60
Oct 31 01:30:20.393817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.393793    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerDied","Data":"a5e6af385dc8cff9edd3a5d7901661ca6f3a9888c03d18efa241eae40e90f902"}
Oct 31 01:30:20.397205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.394472    5419 scope.go:122] "RemoveContainer" containerID="a5e6af385dc8cff9edd3a5d7901661ca6f3a9888c03d18efa241eae40e90f902"
Oct 31 01:30:20.397205 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:20.395053    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5_odh-ai-gateway-infra(906ed670-89e6-42b5-a18d-ec1acad34231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" podUID="906ed670-89e6-42b5-a18d-ec1acad34231"
Oct 31 01:30:20.399331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.399305    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"
Oct 31 01:30:20.399440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.399304    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln" event={"ID":"77b67658-bc01-48df-b1cf-beef150b68ef","Type":"ContainerDied","Data":"c6384aa5b3f3a64ac8406d7331d81956d2f857207af03baa4aa3f74b920bba0d"}
Oct 31 01:30:20.401697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.401393    5419 scope.go:122] "RemoveContainer" containerID="d704575d3e8e8cdfcbac3f6ffc60f49e02cc96e9c58ebcb95e1a389a4cf5cdc9"
Oct 31 01:30:20.425640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.425606    5419 scope.go:122] "RemoveContainer" containerID="65fc8c9b14c51fdf32634841d12000878cadb0eb0200740ba8d4c85c24a9afb4"
Oct 31 01:30:20.593047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.593019    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:20.616411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.616385    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca\") pod \"80a960de-8acf-4679-85e2-b560a9a2c1e8\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") "
Oct 31 01:30:20.616592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.616576    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw\" (UniqueName: \"kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw\") pod \"80a960de-8acf-4679-85e2-b560a9a2c1e8\" (UID: \"80a960de-8acf-4679-85e2-b560a9a2c1e8\") "
Oct 31 01:30:20.616710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.616688    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca" pod "80a960de-8acf-4679-85e2-b560a9a2c1e8" (UID: "80a960de-8acf-4679-85e2-b560a9a2c1e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:20.617308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.617088    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a960de-8acf-4679-85e2-b560a9a2c1e8-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:20.618754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.618732    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw" pod "80a960de-8acf-4679-85e2-b560a9a2c1e8" (UID: "80a960de-8acf-4679-85e2-b560a9a2c1e8"). InnerVolumeSpecName "kube-api-access-n62pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:20.626691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.626674    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:20.718561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.718533    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc\" (UniqueName: \"kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc\") pod \"fd2042ee-34cc-413a-bcf5-ff04ae176627\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") "
Oct 31 01:30:20.718685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.718648    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca\") pod \"fd2042ee-34cc-413a-bcf5-ff04ae176627\" (UID: \"fd2042ee-34cc-413a-bcf5-ff04ae176627\") "
Oct 31 01:30:20.718968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.718944    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca" pod "fd2042ee-34cc-413a-bcf5-ff04ae176627" (UID: "fd2042ee-34cc-413a-bcf5-ff04ae176627"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:20.719222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.719209    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62pw\" (UniqueName: \"kubernetes.io/projected/80a960de-8acf-4679-85e2-b560a9a2c1e8-kube-api-access-n62pw\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:20.719297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.719224    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2042ee-34cc-413a-bcf5-ff04ae176627-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:20.721260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.721235    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc" pod "fd2042ee-34cc-413a-bcf5-ff04ae176627" (UID: "fd2042ee-34cc-413a-bcf5-ff04ae176627"). InnerVolumeSpecName "kube-api-access-f5wlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:20.738008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.737954    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"]
Oct 31 01:30:20.821003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.820974    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5wlc\" (UniqueName: \"kubernetes.io/projected/fd2042ee-34cc-413a-bcf5-ff04ae176627-kube-api-access-f5wlc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:20.840878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.840846    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" path="/var/lib/kubelet/pods/dfa841e8-26f3-4038-a2a8-b1c832ba35f3/volumes"
Oct 31 01:30:20.841516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.841501    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"]
Oct 31 01:30:20.841589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.841533    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890170-vxhln"]
Oct 31 01:30:20.942965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.942933    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"]
Oct 31 01:30:20.946298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:20.946278    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890170-p6xqb"]
Oct 31 01:30:21.408443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.408413    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"
Oct 31 01:30:21.408443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.408423    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf" event={"ID":"fd2042ee-34cc-413a-bcf5-ff04ae176627","Type":"ContainerDied","Data":"f767fbdc625f583fd90298820f27fdcc0a02bcc82aeb7d23e552da07ef53a2ab"}
Oct 31 01:30:21.408967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.408477    5419 scope.go:122] "RemoveContainer" containerID="184d3147fddaed2144ea4562553c6f6d478774266859e541c660f944b25e779c"
Oct 31 01:30:21.411213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.411191    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"
Oct 31 01:30:21.411352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.411218    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849" event={"ID":"80a960de-8acf-4679-85e2-b560a9a2c1e8","Type":"ContainerDied","Data":"ab598776e3a33273a2cbcd85be7ab1cc80cab648b11fe6091d0ed0e4cce47efb"}
Oct 31 01:30:21.419793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.419769    5419 scope.go:122] "RemoveContainer" containerID="c97306cfba6f1ec00afa55f2d6dab8b0ece7e4d763aac3855955303f764c29f9"
Oct 31 01:30:21.593129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.584539    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"]
Oct 31 01:30:21.629919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.629888    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890170-n54gf"]
Oct 31 01:30:21.668909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.668855    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:21.696966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.696940    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"]
Oct 31 01:30:21.700582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.700560    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-01cbd8-29890170-c5849"]
Oct 31 01:30:21.739633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.739608    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca\") pod \"906ed670-89e6-42b5-a18d-ec1acad34231\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") "
Oct 31 01:30:21.739866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.739681    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs\" (UniqueName: \"kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs\") pod \"906ed670-89e6-42b5-a18d-ec1acad34231\" (UID: \"906ed670-89e6-42b5-a18d-ec1acad34231\") "
Oct 31 01:30:21.739971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.739950    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca" pod "906ed670-89e6-42b5-a18d-ec1acad34231" (UID: "906ed670-89e6-42b5-a18d-ec1acad34231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:21.740253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.740237    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/906ed670-89e6-42b5-a18d-ec1acad34231-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:21.741928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.741908    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs" pod "906ed670-89e6-42b5-a18d-ec1acad34231" (UID: "906ed670-89e6-42b5-a18d-ec1acad34231"). InnerVolumeSpecName "kube-api-access-jv7rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:21.788997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.788963    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"]
Oct 31 01:30:21.789214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.789190    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy" containerID="cri-o://d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb" gracePeriod=30
Oct 31 01:30:21.792472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.792449    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"]
Oct 31 01:30:21.792675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.792654    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy" containerID="cri-o://2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c" gracePeriod=30
Oct 31 01:30:21.795801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.795776    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"]
Oct 31 01:30:21.796014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.795990    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy" containerID="cri-o://68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085" gracePeriod=30
Oct 31 01:30:21.841497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:21.841463    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv7rs\" (UniqueName: \"kubernetes.io/projected/906ed670-89e6-42b5-a18d-ec1acad34231-kube-api-access-jv7rs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:22.430159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.430098    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5" event={"ID":"906ed670-89e6-42b5-a18d-ec1acad34231","Type":"ContainerDied","Data":"b75e08473052ae2d3e90d05b90ee9b9abd03ddc8ef09a9eead17894ee475efa0"}
Oct 31 01:30:22.430642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.430238    5419 scope.go:122] "RemoveContainer" containerID="a5e6af385dc8cff9edd3a5d7901661ca6f3a9888c03d18efa241eae40e90f902"
Oct 31 01:30:22.430642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.430415    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"
Oct 31 01:30:22.711660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.711497    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"]
Oct 31 01:30:22.715414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.715388    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890170-5j5d5"]
Oct 31 01:30:22.838348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.838320    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" path="/var/lib/kubelet/pods/68e2f664-bc53-437f-833a-982c2b7f1a21/volumes"
Oct 31 01:30:22.838792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.838779    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" path="/var/lib/kubelet/pods/77b67658-bc01-48df-b1cf-beef150b68ef/volumes"
Oct 31 01:30:22.839181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.839168    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" path="/var/lib/kubelet/pods/80a960de-8acf-4679-85e2-b560a9a2c1e8/volumes"
Oct 31 01:30:22.839867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.839852    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" path="/var/lib/kubelet/pods/906ed670-89e6-42b5-a18d-ec1acad34231/volumes"
Oct 31 01:30:22.840259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:22.840246    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" path="/var/lib/kubelet/pods/fd2042ee-34cc-413a-bcf5-ff04ae176627/volumes"
Oct 31 01:30:27.113228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.105757    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.142851    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.142938    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143012    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143057    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143101    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143216    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143270    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143298    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143340    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.143383    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert\") pod \"1b40557c-6f00-44c2-9332-f94ba4998905\" (UID: \"1b40557c-6f00-44c2-9332-f94ba4998905\") "
Oct 31 01:30:27.144469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.144228    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.146280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.146069    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.146411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.146295    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.146484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.146444    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.146484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.146459    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.149130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.146935    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:27.149130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.147887    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:27.149130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.148877    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "kube-api-access-88s8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.151288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.151261    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token" pod "1b40557c-6f00-44c2-9332-f94ba4998905" (UID: "1b40557c-6f00-44c2-9332-f94ba4998905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.158132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.158110    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:30:27.164270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.164253    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:30:27.245206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245179    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245223    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.245353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245251    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.245353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245288    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245324    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.245549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245353    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.245549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245400    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.245549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245453    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245473    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.245549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245521    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245572    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245649    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245709    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245722    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245733    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245755    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.245829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245822    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.246217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245863    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.246217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245935    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.246217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245960    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token\") pod \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\" (UID: \"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5\") "
Oct 31 01:30:27.246217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.246020    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2\") pod \"82cebf8a-0c3f-470b-81fd-9c0952261b94\" (UID: \"82cebf8a-0c3f-470b-81fd-9c0952261b94\") "
Oct 31 01:30:27.247061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247033    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247066    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247082    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s8p\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-kube-api-access-88s8p\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247098    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247111    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b40557c-6f00-44c2-9332-f94ba4998905-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247126    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247188    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247204    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b40557c-6f00-44c2-9332-f94ba4998905-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247218    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247230    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b40557c-6f00-44c2-9332-f94ba4998905-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247243    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b40557c-6f00-44c2-9332-f94ba4998905-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245684    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.245997    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.246013    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.246209    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.246628    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247073    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:30:27.247528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.247306    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.248562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.248536    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.248672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.248594    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:27.248777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.248752    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.248897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.248785    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:30:27.248969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.248921    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "kube-api-access-jmwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.249344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.249325    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:27.249480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.249467    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2" pod "82cebf8a-0c3f-470b-81fd-9c0952261b94" (UID: "82cebf8a-0c3f-470b-81fd-9c0952261b94"). InnerVolumeSpecName "kube-api-access-fptm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.249642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.249625    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token" pod "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" (UID: "5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348488    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fptm2\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-kube-api-access-fptm2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348516    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348530    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348543    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348552    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348560    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348569    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.348574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348578    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmwtz\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-kube-api-access-jmwtz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348587    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348598    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348622    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82cebf8a-0c3f-470b-81fd-9c0952261b94-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348633    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348641    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348652    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82cebf8a-0c3f-470b-81fd-9c0952261b94-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.349189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.348660    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:27.473766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.473734    5419 generic.go:356] "Generic (PLEG): container finished" podID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerID="68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085" exitCode=0
Oct 31 01:30:27.473955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.473779    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"
Oct 31 01:30:27.473955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.473809    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" event={"ID":"82cebf8a-0c3f-470b-81fd-9c0952261b94","Type":"ContainerDied","Data":"68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085"}
Oct 31 01:30:27.473955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.473892    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2" event={"ID":"82cebf8a-0c3f-470b-81fd-9c0952261b94","Type":"ContainerDied","Data":"689c3af61fea9af9cc6c7ae0a4f9df36b46f38c39cd2b087a1f913c7714c56f2"}
Oct 31 01:30:27.473955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.473929    5419 scope.go:122] "RemoveContainer" containerID="68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085"
Oct 31 01:30:27.475719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.475681    5419 generic.go:356] "Generic (PLEG): container finished" podID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerID="2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c" exitCode=0
Oct 31 01:30:27.475846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.475756    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"
Oct 31 01:30:27.476006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.475816    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" event={"ID":"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5","Type":"ContainerDied","Data":"2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c"}
Oct 31 01:30:27.476104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.476042    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66" event={"ID":"5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5","Type":"ContainerDied","Data":"b1ee93dce81761591b14976bb3fe160e2fd60394fee036e14f44946b3836ef58"}
Oct 31 01:30:27.482201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.482179    5419 generic.go:356] "Generic (PLEG): container finished" podID="1b40557c-6f00-44c2-9332-f94ba4998905" containerID="d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb" exitCode=0
Oct 31 01:30:27.482305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.482232    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"
Oct 31 01:30:27.482305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.482257    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" event={"ID":"1b40557c-6f00-44c2-9332-f94ba4998905","Type":"ContainerDied","Data":"d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb"}
Oct 31 01:30:27.482305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.482300    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx" event={"ID":"1b40557c-6f00-44c2-9332-f94ba4998905","Type":"ContainerDied","Data":"76f17466c8bc5efa8c9d4382d427cccd0ce1d8e5a5b46b4f26fad1c55dcb4bac"}
Oct 31 01:30:27.486003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.485982    5419 scope.go:122] "RemoveContainer" containerID="68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085"
Oct 31 01:30:27.487685 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:27.487656    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085\": container with ID starting with 68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085 not found: ID does not exist" containerID="68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085"
Oct 31 01:30:27.487831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.487803    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085"} err="failed to get container status \"68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085\": rpc error: code = NotFound desc = could not find container \"68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085\": container with ID starting with 68e023d25a1d6673e5bc0c80b24553388cb397bb0b605a4b0d4c14eafbae8085 not found: ID does not exist"
Oct 31 01:30:27.487831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.487831    5419 scope.go:122] "RemoveContainer" containerID="2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c"
Oct 31 01:30:27.498213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.498194    5419 scope.go:122] "RemoveContainer" containerID="2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c"
Oct 31 01:30:27.498555 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:27.498526    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c\": container with ID starting with 2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c not found: ID does not exist" containerID="2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c"
Oct 31 01:30:27.498555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.498563    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c"} err="failed to get container status \"2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c\": rpc error: code = NotFound desc = could not find container \"2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c\": container with ID starting with 2056c18f8625f7df97f15014a4197c004bec5679e8edb565aa63172e0aada56c not found: ID does not exist"
Oct 31 01:30:27.498863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.498586    5419 scope.go:122] "RemoveContainer" containerID="d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb"
Oct 31 01:30:27.540842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.540709    5419 scope.go:122] "RemoveContainer" containerID="d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb"
Oct 31 01:30:27.541353 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:27.541319    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb\": container with ID starting with d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb not found: ID does not exist" containerID="d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb"
Oct 31 01:30:27.541481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.541365    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb"} err="failed to get container status \"d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb\": rpc error: code = NotFound desc = could not find container \"d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb\": container with ID starting with d1b079111f6f56f9901f331e8c24b5898ecd9a11807eab03439f15a5374657fb not found: ID does not exist"
Oct 31 01:30:27.643222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.643130    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"]
Oct 31 01:30:27.646286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.646268    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-66nr2"]
Oct 31 01:30:27.750939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.750908    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"]
Oct 31 01:30:27.756604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.756583    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-mp6mx"]
Oct 31 01:30:27.861279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.861247    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"]
Oct 31 01:30:27.864603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:27.864580    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-jfp66"]
Oct 31 01:30:28.838289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:28.838259    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" path="/var/lib/kubelet/pods/1b40557c-6f00-44c2-9332-f94ba4998905/volumes"
Oct 31 01:30:28.838705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:28.838691    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" path="/var/lib/kubelet/pods/5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5/volumes"
Oct 31 01:30:28.839097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:28.839085    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" path="/var/lib/kubelet/pods/82cebf8a-0c3f-470b-81fd-9c0952261b94/volumes"
Oct 31 01:30:35.255769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:35.255713    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main" containerID="cri-o://525f1a95ba2c7ab15788c239fe656ac7979ee3c9178c98c08e77c6953ecee97c" gracePeriod=45
Oct 31 01:30:35.525277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:35.511816    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:30:35.671700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:35.669971    5419 generic.go:356] "Generic (PLEG): container finished" podID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerID="525f1a95ba2c7ab15788c239fe656ac7979ee3c9178c98c08e77c6953ecee97c" exitCode=0
Oct 31 01:30:35.671700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:35.670202    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" event={"ID":"94b2bd66-5e24-4fea-abb7-0bf82992a828","Type":"ContainerDied","Data":"525f1a95ba2c7ab15788c239fe656ac7979ee3c9178c98c08e77c6953ecee97c"}
Oct 31 01:30:35.958970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:35.958870    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main" probeResult="failure" output="Get \"https://10.133.0.114:8000/ready\": dial tcp 10.133.0.114:8000: connect: connection refused"
Oct 31 01:30:36.529301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.529278    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:30:36.561944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.561909    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir\") pod \"94b2bd66-5e24-4fea-abb7-0bf82992a828\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") "
Oct 31 01:30:36.562154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562031    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8\" (UniqueName: \"kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8\") pod \"94b2bd66-5e24-4fea-abb7-0bf82992a828\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") "
Oct 31 01:30:36.562154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562069    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm\") pod \"94b2bd66-5e24-4fea-abb7-0bf82992a828\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") "
Oct 31 01:30:36.562154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562113    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache\") pod \"94b2bd66-5e24-4fea-abb7-0bf82992a828\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") "
Oct 31 01:30:36.562368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562167    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs\") pod \"94b2bd66-5e24-4fea-abb7-0bf82992a828\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") "
Oct 31 01:30:36.562368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562212    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home\") pod \"94b2bd66-5e24-4fea-abb7-0bf82992a828\" (UID: \"94b2bd66-5e24-4fea-abb7-0bf82992a828\") "
Oct 31 01:30:36.562368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562260    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir" pod "94b2bd66-5e24-4fea-abb7-0bf82992a828" (UID: "94b2bd66-5e24-4fea-abb7-0bf82992a828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:36.562554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562526    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache" pod "94b2bd66-5e24-4fea-abb7-0bf82992a828" (UID: "94b2bd66-5e24-4fea-abb7-0bf82992a828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:36.562788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562765    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home" pod "94b2bd66-5e24-4fea-abb7-0bf82992a828" (UID: "94b2bd66-5e24-4fea-abb7-0bf82992a828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:36.563030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.562983    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:36.563030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.563013    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:36.563030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.563028    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:36.565603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.565578    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs" pod "94b2bd66-5e24-4fea-abb7-0bf82992a828" (UID: "94b2bd66-5e24-4fea-abb7-0bf82992a828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:30:36.565724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.565602    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8" pod "94b2bd66-5e24-4fea-abb7-0bf82992a828" (UID: "94b2bd66-5e24-4fea-abb7-0bf82992a828"). InnerVolumeSpecName "kube-api-access-562n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:30:36.567252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.567226    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm" pod "94b2bd66-5e24-4fea-abb7-0bf82992a828" (UID: "94b2bd66-5e24-4fea-abb7-0bf82992a828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:30:36.668427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.668388    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94b2bd66-5e24-4fea-abb7-0bf82992a828-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:36.668427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.668429    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-562n8\" (UniqueName: \"kubernetes.io/projected/94b2bd66-5e24-4fea-abb7-0bf82992a828-kube-api-access-562n8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:36.668639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.668444    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94b2bd66-5e24-4fea-abb7-0bf82992a828-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:30:36.696162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.685347    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw" event={"ID":"94b2bd66-5e24-4fea-abb7-0bf82992a828","Type":"ContainerDied","Data":"8b53a255b2588558738befd87e8ac7c7e141795dbaab1381f6ffb313a2e9c735"}
Oct 31 01:30:36.696162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.685421    5419 scope.go:122] "RemoveContainer" containerID="525f1a95ba2c7ab15788c239fe656ac7979ee3c9178c98c08e77c6953ecee97c"
Oct 31 01:30:36.696162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.685587    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"
Oct 31 01:30:36.861070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.860892    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"]
Oct 31 01:30:36.864275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:36.864247    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-18bf762c/e2e-lifecycle-model-18bf762c-kserve-58d54864df-8ktqw"]
Oct 31 01:30:38.839417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:38.839346    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" path="/var/lib/kubelet/pods/94b2bd66-5e24-4fea-abb7-0bf82992a828/volumes"
Oct 31 01:30:39.457778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.457751    5419 scope.go:122] "RemoveContainer" containerID="0d1a4b007261f48e68d0a430908b63083891e6c256226b6355b0dae623f64666"
Oct 31 01:30:39.468121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.468103    5419 scope.go:122] "RemoveContainer" containerID="a27f4f273d4c3149d3b7dac4f574a566568b541b0ad6993db754e95197172f97"
Oct 31 01:30:39.481550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.481440    5419 scope.go:122] "RemoveContainer" containerID="2438425c89a245889ed764b4d234cbda05a66ef1442ba3cdf8236b3ffa5b900c"
Oct 31 01:30:39.510777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.510750    5419 scope.go:122] "RemoveContainer" containerID="1dd07cbf9b11c917b2bce6b9caf58e4a7ddc53588ee9b1a7ae4c68c9d40cceeb"
Oct 31 01:30:39.520366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.520345    5419 scope.go:122] "RemoveContainer" containerID="d9bded28056bd91ebb6c193f04d0e71510fb42d2db98372072804d282c4d90bf"
Oct 31 01:30:39.529403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.529385    5419 scope.go:122] "RemoveContainer" containerID="38454dc3d9f053b11ba594812650bd11a35f379a750ec8910a251b75d157c46b"
Oct 31 01:30:39.539344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.539327    5419 scope.go:122] "RemoveContainer" containerID="b8633c3092045224e7bce0e09e7305b4388d08ec2561dc31f8bd25f82272e48b"
Oct 31 01:30:39.548174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.548128    5419 scope.go:122] "RemoveContainer" containerID="0bce03a847a34c27887ad75d78a481861483a3b4e9c393422593063702e749a9"
Oct 31 01:30:39.557299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.557279    5419 scope.go:122] "RemoveContainer" containerID="7a9993cf25575dac08ad7ceb3fa3fb890b6cadbafe4d27ecacbd86c6ca6bc9c7"
Oct 31 01:30:39.567440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.567423    5419 scope.go:122] "RemoveContainer" containerID="68dd5f108222d348b7fe47d3cb9d0c02abbe3e6fe2a9a20522b72ef76e799412"
Oct 31 01:30:39.601095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:39.601075    5419 scope.go:122] "RemoveContainer" containerID="89316f23e308ff008a78fa5be6cdcc17889ebe68995d10fe70b6f6d9c8f9625a"
Oct 31 01:30:42.048448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:42.048417    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"]
Oct 31 01:30:42.098084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:42.098052    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87266d5-pkfxj"]
Oct 31 01:30:42.838834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:42.838789    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6c0141-26a9-473b-a1a2-7a26ba0ee952" path="/var/lib/kubelet/pods/fd6c0141-26a9-473b-a1a2-7a26ba0ee952/volumes"
Oct 31 01:30:44.583001 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:30:44.582963    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:30:46.783223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:46.783185    5419 generic.go:356] "Generic (PLEG): container finished" podID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerID="85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be" exitCode=0
Oct 31 01:30:46.783688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:46.783256    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" event={"ID":"1503f6f8-f1f6-4b40-a806-0657fc0da67b","Type":"ContainerDied","Data":"85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"}
Oct 31 01:30:46.783740 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:46.783721    5419 scope.go:122] "RemoveContainer" containerID="85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"
Oct 31 01:30:46.785344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:46.785314    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:30:46.785467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:46.785397    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:30:46.785467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:46.785449    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:30:47.807387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:47.807352    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" event={"ID":"1503f6f8-f1f6-4b40-a806-0657fc0da67b","Type":"ContainerStarted","Data":"073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674"}
Oct 31 01:30:48.317606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:48.317573    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:30:48.318755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:48.318725    5419 patch_prober.go:29] interesting pod/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:30:48.318894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:48.318776    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:30:48.718352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:48.718264    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:30:49.317822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:49.317786    5419 patch_prober.go:29] interesting pod/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:30:49.318268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:49.317846    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:30:50.317698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:50.317665    5419 patch_prober.go:29] interesting pod/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:30:50.317890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:50.317722    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:30:51.318417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:51.318381    5419 patch_prober.go:29] interesting pod/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:30:51.318896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:51.318438    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:30:52.317614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:52.317572    5419 patch_prober.go:29] interesting pod/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 01:30:52.317832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:52.317659    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 01:30:53.322239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:53.322210    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:30:53.323400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:30:53.323382    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:31:00.080682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.080650    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"]
Oct 31 01:31:00.081276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081262    5419 cpu_manager.go:414] "containerMap: removing container" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081278    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081287    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081292    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081301    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081306    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081315    5419 cpu_manager.go:414] "containerMap: removing container" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081319    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081330    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081334    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081343    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081347    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy"
Oct 31 01:31:00.081348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081354    5419 cpu_manager.go:414] "containerMap: removing container" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081360    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081367    5419 cpu_manager.go:414] "containerMap: removing container" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081372    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081388    5419 cpu_manager.go:414] "containerMap: removing container" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081392    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081398    5419 cpu_manager.go:414] "containerMap: removing container" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081403    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081410    5419 cpu_manager.go:414] "containerMap: removing container" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081415    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081425    5419 cpu_manager.go:414] "containerMap: removing container" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081429    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081435    5419 cpu_manager.go:414] "containerMap: removing container" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081440    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081447    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081451    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081458    5419 cpu_manager.go:414] "containerMap: removing container" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081461    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081474    5419 cpu_manager.go:414] "containerMap: removing container" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081479    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081489    5419 cpu_manager.go:414] "containerMap: removing container" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081493    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081577    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081584    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="82cebf8a-0c3f-470b-81fd-9c0952261b94" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081590    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b40557c-6f00-44c2-9332-f94ba4998905" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081595    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081601    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081606    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081612    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081616    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081621    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081627    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081634    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081639    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081644    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081651    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081657    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081662    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c77cdb9-63e8-4646-8c8a-1bdd8fcb2ff5" containerName="istio-proxy"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081669    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081675    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081681    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b2bd66-5e24-4fea-abb7-0bf82992a828" containerName="main"
Oct 31 01:31:00.081725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.081685    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:00.085191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.085173    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.143554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.143516    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bc2046-openshift-default-dockercfg-g4pls\""
Oct 31 01:31:00.196558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.190015    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.233375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.233334    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.233680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.233660    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.233874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.233858    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.235985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.235958    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n5tg\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.236298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.236274    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.236503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.236485    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.236814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.236771    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.236962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.236946    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.239045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.239024    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"]
Oct 31 01:31:00.349585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.341093    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354223    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354324    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354409    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354477    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354582    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354635    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n5tg\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354700    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.354772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.354732    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.355126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.351886    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.355426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.355404    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.356282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.355912    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.356282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.356122    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.370174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.368625    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.455332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.454371    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.455332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.454765    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.455607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.455402    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n5tg\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.455829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.455804    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy\") pod \"e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") " pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:00.709827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:00.709744    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:01.137443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:01.137417    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"]
Oct 31 01:31:01.138167 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:01.138116    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4228b72_6d15_46f9_a069_c68007a190a9.slice/crio-331adc4e3fbea7943c77dbb9403c354c4bfe792afa7f690952d325496ce93a9b WatchSource:0}: Error finding container 331adc4e3fbea7943c77dbb9403c354c4bfe792afa7f690952d325496ce93a9b: Status 404 returned error can't find the container with id 331adc4e3fbea7943c77dbb9403c354c4bfe792afa7f690952d325496ce93a9b
Oct 31 01:31:01.140745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:01.140713    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:01.140825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:01.140778    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:01.140825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:01.140805    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:02.044102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:02.044065    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" event={"ID":"e4228b72-6d15-46f9-a069-c68007a190a9","Type":"ContainerStarted","Data":"47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"}
Oct 31 01:31:02.044304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:02.044113    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" event={"ID":"e4228b72-6d15-46f9-a069-c68007a190a9","Type":"ContainerStarted","Data":"331adc4e3fbea7943c77dbb9403c354c4bfe792afa7f690952d325496ce93a9b"}
Oct 31 01:31:02.211869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:02.211815    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podStartSLOduration=2.211799477 podStartE2EDuration="2.211799477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:02.209867384 +0000 UTC m=+1583.510823957" watchObservedRunningTime="2026-10-31 01:31:02.211799477 +0000 UTC m=+1583.512756054"
Oct 31 01:31:02.710521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:02.710496    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:02.715792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:02.715764    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:03.103215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:03.103182    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:03.104677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:03.104571    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:31:03.284856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:03.284816    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"]
Oct 31 01:31:03.285371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:03.285114    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy" containerID="cri-o://415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2" gracePeriod=30
Oct 31 01:31:04.952027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:04.951992    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:31:08.714617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.714588    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.811612    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.811710    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.811762    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.811807    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.811873    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.811974    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.812007    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.812101    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.821214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.812309    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token\") pod \"e911c5ac-ba30-4372-8da0-7290fd15838c\" (UID: \"e911c5ac-ba30-4372-8da0-7290fd15838c\") "
Oct 31 01:31:08.855576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.855246    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.862634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.862424    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.863448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.863413    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.867161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.863690    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:08.876376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.874384    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "kube-api-access-wf7d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.965241    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.967794    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.967947    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e911c5ac-ba30-4372-8da0-7290fd15838c-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.967984    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.968010    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.968032    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.968051    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.968066    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.970173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.968080    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf7d9\" (UniqueName: \"kubernetes.io/projected/e911c5ac-ba30-4372-8da0-7290fd15838c-kube-api-access-wf7d9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:08.984326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:08.984263    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:09.124423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.069121    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy" pod "e911c5ac-ba30-4372-8da0-7290fd15838c" (UID: "e911c5ac-ba30-4372-8da0-7290fd15838c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:09.124423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.069665    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:09.124423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.069693    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e911c5ac-ba30-4372-8da0-7290fd15838c-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:09.342167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.319324    5419 generic.go:356] "Generic (PLEG): container finished" podID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerID="415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2" exitCode=0
Oct 31 01:31:09.342167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.319499    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" event={"ID":"e911c5ac-ba30-4372-8da0-7290fd15838c","Type":"ContainerDied","Data":"415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2"}
Oct 31 01:31:09.342167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.319572    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c" event={"ID":"e911c5ac-ba30-4372-8da0-7290fd15838c","Type":"ContainerDied","Data":"a6ceb44184ceb396eb185deaf416f70bc5a356ad385ac29931a93a54f385b2dd"}
Oct 31 01:31:09.342167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.319739    5419 scope.go:122] "RemoveContainer" containerID="415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2"
Oct 31 01:31:09.342167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.319802    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"
Oct 31 01:31:09.552160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.542653    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"]
Oct 31 01:31:09.557683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.553328    5419 cpu_manager.go:414] "containerMap: removing container" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:09.557899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.557875    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:09.558013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.558003    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerName="cleanup"
Oct 31 01:31:09.584017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.583984    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a1d8b5-8f22-4a71-aa95-61842e00abf8" containerName="cleanup"
Oct 31 01:31:09.586311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586287    5419 cpu_manager.go:414] "containerMap: removing container" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:09.586496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586480    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e2f664-bc53-437f-833a-982c2b7f1a21" containerName="cleanup"
Oct 31 01:31:09.586606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586595    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:09.586683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586673    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:09.586783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586773    5419 cpu_manager.go:414] "containerMap: removing container" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:09.586870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586860    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:09.586964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.586953    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy"
Oct 31 01:31:09.587046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.587036    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy"
Oct 31 01:31:09.587195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.587125    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy"
Oct 31 01:31:09.614684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.614654    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy"
Oct 31 01:31:09.614867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.614854    5419 cpu_manager.go:414] "containerMap: removing container" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:09.614962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.614951    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906ed670-89e6-42b5-a18d-ec1acad34231" containerName="cleanup"
Oct 31 01:31:09.615049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.615039    5419 cpu_manager.go:414] "containerMap: removing container" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:09.615123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.615113    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:09.615472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.615453    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy"
Oct 31 01:31:09.615607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.615595    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" containerName="istio-proxy"
Oct 31 01:31:09.633238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.633203    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a960de-8acf-4679-85e2-b560a9a2c1e8" containerName="cleanup"
Oct 31 01:31:09.633476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.633457    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b67658-bc01-48df-b1cf-beef150b68ef" containerName="cleanup"
Oct 31 01:31:09.633605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.633592    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa841e8-26f3-4038-a2a8-b1c832ba35f3" containerName="cleanup"
Oct 31 01:31:09.633704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.633694    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2042ee-34cc-413a-bcf5-ff04ae176627" containerName="cleanup"
Oct 31 01:31:09.690182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.642656    5419 scope.go:122] "RemoveContainer" containerID="7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"
Oct 31 01:31:09.690182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.687907    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"]
Oct 31 01:31:09.690182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.688038    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:09.764193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.746827    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"]
Oct 31 01:31:09.764193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.764085    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-18bf762c-openshift-default-77b9fdd9b5-vf89c"]
Oct 31 01:31:09.764193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.760845    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8trm\" (UniqueName: \"kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm\") pod \"payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:09.765000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.764425    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:09.765752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.765677    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bc2046\""
Oct 31 01:31:09.765979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.765940    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bc2046-dockercfg-qk4d6\""
Oct 31 01:31:09.806219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.803123    5419 scope.go:122] "RemoveContainer" containerID="415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2"
Oct 31 01:31:09.806219 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:31:09.803981    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2\": container with ID starting with 415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2 not found: ID does not exist" containerID="415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2"
Oct 31 01:31:09.806219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.804017    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2"} err="failed to get container status \"415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2\": rpc error: code = NotFound desc = could not find container \"415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2\": container with ID starting with 415d7124bebb5cef3b3940f765777f4e20445d67e9c1a5b516e24e26534b34b2 not found: ID does not exist"
Oct 31 01:31:09.806219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.804042    5419 scope.go:122] "RemoveContainer" containerID="7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"
Oct 31 01:31:09.806219 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:31:09.804635    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44\": container with ID starting with 7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44 not found: ID does not exist" containerID="7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"
Oct 31 01:31:09.806219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.804666    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44"} err="failed to get container status \"7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44\": rpc error: code = NotFound desc = could not find container \"7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44\": container with ID starting with 7a41f0d773a78bad518678ddc374a611664573a020cc00b1f8a0c47548afee44 not found: ID does not exist"
Oct 31 01:31:09.865522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.865493    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8trm\" (UniqueName: \"kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm\") pod \"payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:09.865680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.865574    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:09.866243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.866226    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:09.922959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:09.922926    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8trm\" (UniqueName: \"kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm\") pod \"payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:10.052450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:10.052412    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:10.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:10.454286    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"]
Oct 31 01:31:10.454899 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:10.454875    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab31518_f838_4f11_a6fa_0c64f28ae9cb.slice/crio-053fed798502f8d941372920424098a2799d4818cb1d67abb18b2c8982ffaada WatchSource:0}: Error finding container 053fed798502f8d941372920424098a2799d4818cb1d67abb18b2c8982ffaada: Status 404 returned error can't find the container with id 053fed798502f8d941372920424098a2799d4818cb1d67abb18b2c8982ffaada
Oct 31 01:31:10.838949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:10.838913    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e911c5ac-ba30-4372-8da0-7290fd15838c" path="/var/lib/kubelet/pods/e911c5ac-ba30-4372-8da0-7290fd15838c/volumes"
Oct 31 01:31:11.337631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:11.337588    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg" event={"ID":"0ab31518-f838-4f11-a6fa-0c64f28ae9cb","Type":"ContainerStarted","Data":"c66c787b0d219710afb1cb16a6ed20f6dbf8394ed13b1abdacbde4780014c2e6"}
Oct 31 01:31:11.337838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:11.337651    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:11.337838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:11.337668    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg" event={"ID":"0ab31518-f838-4f11-a6fa-0c64f28ae9cb","Type":"ContainerStarted","Data":"053fed798502f8d941372920424098a2799d4818cb1d67abb18b2c8982ffaada"}
Oct 31 01:31:11.455958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:11.455887    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg" podStartSLOduration=2.455873291 podStartE2EDuration="2.455873291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:11.455645339 +0000 UTC m=+1592.756601921" watchObservedRunningTime="2026-10-31 01:31:11.455873291 +0000 UTC m=+1592.756829868"
Oct 31 01:31:13.893264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.893233    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"]
Oct 31 01:31:13.897677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.897660    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:13.954621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.954589    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"]
Oct 31 01:31:13.963776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.963743    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:13.964086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.964060    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8b6\" (UniqueName: \"kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:13.964219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.964180    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:13.999031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.999002    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:31:13.999031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.999016    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:31:13.999230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.999040    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:31:13.999230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:13.999093    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:31:14.065995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.065966    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.066185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.066032    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.066248 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.066218    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8b6\" (UniqueName: \"kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.066847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.066762    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.068691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.068667    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.123019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.122986    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8b6\" (UniqueName: \"kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6\") pod \"maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.211908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.211828    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:14.584405 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:31:14.584374    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:31:14.689457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:14.689430    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"]
Oct 31 01:31:14.690288 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:14.690262    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23326a5_fbea_49b6_81cf_28d29848cd16.slice/crio-c2b1b5149c94bafdb4584c9d0c6aef0ce1932c911bf281333cb8b2c1c69cb184 WatchSource:0}: Error finding container c2b1b5149c94bafdb4584c9d0c6aef0ce1932c911bf281333cb8b2c1c69cb184: Status 404 returned error can't find the container with id c2b1b5149c94bafdb4584c9d0c6aef0ce1932c911bf281333cb8b2c1c69cb184
Oct 31 01:31:15.356416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:15.355558    5419 patch_prober.go:29] interesting pod/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": read tcp 10.133.0.2:56548->10.133.0.129:15021: read: connection reset by peer" start-of-body=
Oct 31 01:31:15.356416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:15.355639    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": read tcp 10.133.0.2:56548->10.133.0.129:15021: read: connection reset by peer"
Oct 31 01:31:15.374372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:15.374323    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk" event={"ID":"b23326a5-fbea-49b6-81cf-28d29848cd16","Type":"ContainerStarted","Data":"544f4a8030848cc407fad98bb7fd66272c62a9e467d9c4a779f48c25760d399f"}
Oct 31 01:31:15.374525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:15.374431    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk" event={"ID":"b23326a5-fbea-49b6-81cf-28d29848cd16","Type":"ContainerStarted","Data":"c2b1b5149c94bafdb4584c9d0c6aef0ce1932c911bf281333cb8b2c1c69cb184"}
Oct 31 01:31:15.488890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:15.488836    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk" podStartSLOduration=2.48882303 podStartE2EDuration="2.48882303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:15.487987737 +0000 UTC m=+1596.788944314" watchObservedRunningTime="2026-10-31 01:31:15.48882303 +0000 UTC m=+1596.789779641"
Oct 31 01:31:16.382867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.382836    5419 generic.go:356] "Generic (PLEG): container finished" podID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerID="752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271" exitCode=0
Oct 31 01:31:16.383323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.382911    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" event={"ID":"5073b593-a911-413c-8c9f-22226ee3e0f7","Type":"ContainerDied","Data":"752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"}
Oct 31 01:31:16.383493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.383474    5419 scope.go:122] "RemoveContainer" containerID="752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"
Oct 31 01:31:16.384586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.384566    5419 generic.go:356] "Generic (PLEG): container finished" podID="b23326a5-fbea-49b6-81cf-28d29848cd16" containerID="544f4a8030848cc407fad98bb7fd66272c62a9e467d9c4a779f48c25760d399f" exitCode=0
Oct 31 01:31:16.384672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.384617    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk" event={"ID":"b23326a5-fbea-49b6-81cf-28d29848cd16","Type":"ContainerDied","Data":"544f4a8030848cc407fad98bb7fd66272c62a9e467d9c4a779f48c25760d399f"}
Oct 31 01:31:16.385273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.385245    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:16.385376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.385332    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:16.385440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:16.385381    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:17.405842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.405809    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" event={"ID":"5073b593-a911-413c-8c9f-22226ee3e0f7","Type":"ContainerStarted","Data":"3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c"}
Oct 31 01:31:17.502200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.499586    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:31:17.503183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.502542    5419 patch_prober.go:29] interesting pod/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 01:31:17.503183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.502601    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 01:31:17.610341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.610311    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:17.722992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.722968    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca\") pod \"b23326a5-fbea-49b6-81cf-28d29848cd16\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") "
Oct 31 01:31:17.723224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.723021    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6\" (UniqueName: \"kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6\") pod \"b23326a5-fbea-49b6-81cf-28d29848cd16\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") "
Oct 31 01:31:17.723540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.723401    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca\") pod \"b23326a5-fbea-49b6-81cf-28d29848cd16\" (UID: \"b23326a5-fbea-49b6-81cf-28d29848cd16\") "
Oct 31 01:31:17.723681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.723560    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca" pod "b23326a5-fbea-49b6-81cf-28d29848cd16" (UID: "b23326a5-fbea-49b6-81cf-28d29848cd16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:17.724858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.724584    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.726307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.726280    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca" pod "b23326a5-fbea-49b6-81cf-28d29848cd16" (UID: "b23326a5-fbea-49b6-81cf-28d29848cd16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:31:17.726417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.726399    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6" pod "b23326a5-fbea-49b6-81cf-28d29848cd16" (UID: "b23326a5-fbea-49b6-81cf-28d29848cd16"). InnerVolumeSpecName "kube-api-access-2x8b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:17.825900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.825859    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b23326a5-fbea-49b6-81cf-28d29848cd16-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:17.825900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:17.825893    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x8b6\" (UniqueName: \"kubernetes.io/projected/b23326a5-fbea-49b6-81cf-28d29848cd16-kube-api-access-2x8b6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:18.415365 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:18.415338    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"
Oct 31 01:31:18.415841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:18.415405    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk" event={"ID":"b23326a5-fbea-49b6-81cf-28d29848cd16","Type":"ContainerDied","Data":"c2b1b5149c94bafdb4584c9d0c6aef0ce1932c911bf281333cb8b2c1c69cb184"}
Oct 31 01:31:18.415841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:18.415495    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2b1b5149c94bafdb4584c9d0c6aef0ce1932c911bf281333cb8b2c1c69cb184"
Oct 31 01:31:18.499269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:18.499230    5419 patch_prober.go:29] interesting pod/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 01:31:18.499476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:18.499335    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 01:31:19.500484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:19.499965    5419 patch_prober.go:29] interesting pod/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 01:31:19.500484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:19.500037    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 01:31:20.497582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:20.497539    5419 patch_prober.go:29] interesting pod/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 01:31:20.497867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:20.497838    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 01:31:21.499553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:21.499520    5419 patch_prober.go:29] interesting pod/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Oct 31 01:31:21.500000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:21.499585    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Oct 31 01:31:22.347531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:22.347503    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:31:22.563012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:22.562983    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:31:22.563767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:22.563741    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:31:22.564693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:22.564673    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:31:27.579298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:27.579247    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:31:27.905224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:27.904227    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main" probeResult="failure" output="Get \"https://10.133.0.113:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:31:27.927160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:27.926212    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:31:28.197751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.197658    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"]
Oct 31 01:31:28.541566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.541523    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" containerID="cri-o://e8c66f3d10ad61f1f3ce807b2f0a2765479aea0f42ac4381dfc4802f63c36c7d" gracePeriod=30
Oct 31 01:31:28.548374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.548330    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"]
Oct 31 01:31:28.548961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.548935    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy" containerID="cri-o://b5981b42a6b42e68b4302d30b04e9f3ae4ac7ae779491d589981d7383db57365" gracePeriod=30
Oct 31 01:31:28.591303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.591239    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:31:28.592319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.592294    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:31:28.636220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.636189    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"]
Oct 31 01:31:28.636724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.636693    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" containerID="cri-o://3ad73901e95a45ec3ddd583bf3b2be84a86023d4c91a590e718a5dbef00e0937" gracePeriod=30
Oct 31 01:31:28.704041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.704009    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"]
Oct 31 01:31:28.710633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.710595    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" containerID="cri-o://4abdd112adfe55d4ef7b9f82b7462f208a3256e7aff7d3f1a89d4db41124b993" gracePeriod=30
Oct 31 01:31:28.890223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:28.887759    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 01:31:29.624659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.624621    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"]
Oct 31 01:31:29.632167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.626263    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b23326a5-fbea-49b6-81cf-28d29848cd16" containerName="revoke-keys"
Oct 31 01:31:29.633251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.633206    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b23326a5-fbea-49b6-81cf-28d29848cd16" containerName="revoke-keys"
Oct 31 01:31:29.633694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.633673    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b23326a5-fbea-49b6-81cf-28d29848cd16" containerName="revoke-keys"
Oct 31 01:31:29.647552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.647527    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.793356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.785419    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"]
Oct 31 01:31:29.794066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.794031    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-57ea84-openshift-default-dockercfg-7jmzm\""
Oct 31 01:31:29.819024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.818856    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.819497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.819451    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.819861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.819841    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.820160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.820121    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.820398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.820361    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.824160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.824118    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.824446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.824397    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4hq\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.824694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.824673    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:29.824908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:29.824891    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.093104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.093058    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.118284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.093719    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4hq\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.118284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.093820    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.146518    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.146595    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.146735    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.146877    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.146948    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.147042    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.147996    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.148939    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.149451    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.149815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.149703    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.150540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.149898    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.213457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.213417    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.214568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.214536    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.233347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.233311    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.234185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.234118    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4hq\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq\") pod \"e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") " pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.276479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.276446    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:30.713634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.713597    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"]
Oct 31 01:31:30.714397 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:30.714371    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f200f95_279c_4bde_8336_d3975396e88b.slice/crio-3a3fea3a46996dd4240d7c1fc837b1ac4f21d25f10b144c5ac6c8cccb93aa673 WatchSource:0}: Error finding container 3a3fea3a46996dd4240d7c1fc837b1ac4f21d25f10b144c5ac6c8cccb93aa673: Status 404 returned error can't find the container with id 3a3fea3a46996dd4240d7c1fc837b1ac4f21d25f10b144c5ac6c8cccb93aa673
Oct 31 01:31:30.717556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.717522    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:30.717633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.717591    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:30.717633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.717620    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:31:30.965449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.965348    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" event={"ID":"1f200f95-279c-4bde-8336-d3975396e88b","Type":"ContainerStarted","Data":"d5ec4f1e5cbe88e7e69ba5a8505ef85d81a139b89067443ae0c847a6f34bfc3e"}
Oct 31 01:31:30.965449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:30.965420    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" event={"ID":"1f200f95-279c-4bde-8336-d3975396e88b","Type":"ContainerStarted","Data":"3a3fea3a46996dd4240d7c1fc837b1ac4f21d25f10b144c5ac6c8cccb93aa673"}
Oct 31 01:31:31.082893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:31.082840    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podStartSLOduration=2.082824616 podStartE2EDuration="2.082824616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:31.081092456 +0000 UTC m=+1612.382049033" watchObservedRunningTime="2026-10-31 01:31:31.082824616 +0000 UTC m=+1612.383781611"
Oct 31 01:31:31.309113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:31.307171    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:31.331543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:31.331486    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:31.979990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:31.979960    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:31.980943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:31.980918    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:31:32.517592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:32.517561    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:31:32.517749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:32.517614    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:31:33.408324 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:33.408288    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:31:33.408817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:33.408348    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:31:33.442197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:33.442128    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:31:33.442349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:33.442235    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:31:34.075733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.074584    5419 generic.go:356] "Generic (PLEG): container finished" podID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerID="4abdd112adfe55d4ef7b9f82b7462f208a3256e7aff7d3f1a89d4db41124b993" exitCode=0
Oct 31 01:31:34.075733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.074674    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" event={"ID":"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f","Type":"ContainerDied","Data":"4abdd112adfe55d4ef7b9f82b7462f208a3256e7aff7d3f1a89d4db41124b993"}
Oct 31 01:31:34.078504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.078288    5419 generic.go:356] "Generic (PLEG): container finished" podID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerID="e8c66f3d10ad61f1f3ce807b2f0a2765479aea0f42ac4381dfc4802f63c36c7d" exitCode=0
Oct 31 01:31:34.078504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.078429    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" event={"ID":"bb32e360-3c70-43bd-a19a-86d25a2d0d3e","Type":"ContainerDied","Data":"e8c66f3d10ad61f1f3ce807b2f0a2765479aea0f42ac4381dfc4802f63c36c7d"}
Oct 31 01:31:34.080966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.080932    5419 generic.go:356] "Generic (PLEG): container finished" podID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerID="3ad73901e95a45ec3ddd583bf3b2be84a86023d4c91a590e718a5dbef00e0937" exitCode=0
Oct 31 01:31:34.081074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.081050    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" event={"ID":"ad5493b4-7690-4d3a-8b69-4f8105e82abc","Type":"ContainerDied","Data":"3ad73901e95a45ec3ddd583bf3b2be84a86023d4c91a590e718a5dbef00e0937"}
Oct 31 01:31:34.081167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.081102    5419 scope.go:122] "RemoveContainer" containerID="8b8c5a2a157745558a1b41f3cab5f9c4a1033262e5655c84f418743c7643110e"
Oct 31 01:31:34.085709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.085607    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" event={"ID":"b5b6bbbf-5f94-4689-89d5-fb54ff093167","Type":"ContainerDied","Data":"b5981b42a6b42e68b4302d30b04e9f3ae4ac7ae779491d589981d7383db57365"}
Oct 31 01:31:34.085825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.085748    5419 generic.go:356] "Generic (PLEG): container finished" podID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerID="b5981b42a6b42e68b4302d30b04e9f3ae4ac7ae779491d589981d7383db57365" exitCode=0
Oct 31 01:31:34.167578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.166733    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205027    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205166    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205231    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205322    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205424    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205448    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205488    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205533    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.207119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.205609    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket\") pod \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\" (UID: \"bb32e360-3c70-43bd-a19a-86d25a2d0d3e\") "
Oct 31 01:31:34.210267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.209573    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.210267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.209789    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.210267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.209987    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.211104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.210878    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:34.212732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.211293    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.218691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.218652    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.219462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.219413    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:34.229042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.228989    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "kube-api-access-k792g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.250759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.250711    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy" pod "bb32e360-3c70-43bd-a19a-86d25a2d0d3e" (UID: "bb32e360-3c70-43bd-a19a-86d25a2d0d3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.260129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.260103    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.307545    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.307658    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.307700    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.307846    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.307920    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.307953    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308027    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308086    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308229    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token\") pod \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\" (UID: \"b5b6bbbf-5f94-4689-89d5-fb54ff093167\") "
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308888    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k792g\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-kube-api-access-k792g\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308912    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308926    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308941    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308956    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308970    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308983    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.308997    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.309164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.309010    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb32e360-3c70-43bd-a19a-86d25a2d0d3e-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.311216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.310483    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.311216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.310713    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.311216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.310940    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:34.311563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.311351    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.311563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.311517    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.314118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.314081    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.318274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.318248    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.318377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.318320    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "kube-api-access-2vjfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.319427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.319404    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo" pod "b5b6bbbf-5f94-4689-89d5-fb54ff093167" (UID: "b5b6bbbf-5f94-4689-89d5-fb54ff093167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:34.342499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.342407    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:31:34.401470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.401426    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:31:34.409698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409673    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409803    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409829    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409879    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409894    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409953    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.409971    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410013    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410081    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410076    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.410458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410116    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.410458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410319    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:34.410458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410367    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket\") pod \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\" (UID: \"ad5493b4-7690-4d3a-8b69-4f8105e82abc\") "
Oct 31 01:31:34.410736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.410672    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.411424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411399    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411427    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411444    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411458    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411473    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411486    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411500    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411513    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411527    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411540    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.411557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411552    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.412108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411566    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b6bbbf-5f94-4689-89d5-fb54ff093167-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.412108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411580    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b6bbbf-5f94-4689-89d5-fb54ff093167-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.412108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.411593    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjfc\" (UniqueName: \"kubernetes.io/projected/b5b6bbbf-5f94-4689-89d5-fb54ff093167-kube-api-access-2vjfc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.412587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.412561    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.412853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.412834    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.412928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.412848    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:34.412928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.412870    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66" pod "ad5493b4-7690-4d3a-8b69-4f8105e82abc" (UID: "ad5493b4-7690-4d3a-8b69-4f8105e82abc"). InnerVolumeSpecName "kube-api-access-g6d66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.513079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513035    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513099    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513263    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513367    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513405    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513464    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513509    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.513729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513626    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513683    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513716    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy\") pod \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\" (UID: \"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f\") "
Oct 31 01:31:34.513972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.513764    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.514359    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.514386    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.514418    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.514432    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5493b4-7690-4d3a-8b69-4f8105e82abc-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.514446    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.514460    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6d66\" (UniqueName: \"kubernetes.io/projected/ad5493b4-7690-4d3a-8b69-4f8105e82abc-kube-api-access-g6d66\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.515109    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.515294    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.515334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.515313    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:34.516611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.516577    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:31:34.517215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.517191    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:34.518226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.518195    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "kube-api-access-knc2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.521188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.521163    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token" pod "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" (UID: "8f49f690-e0bd-43d5-bb48-2b45feaf2a4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:34.615641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615597    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knc2g\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-kube-api-access-knc2g\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.615641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615637    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.615820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615651    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.615820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615663    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.615820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615677    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.615820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615689    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:34.615820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:34.615701    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:35.094705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.094677    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"
Oct 31 01:31:35.094705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.094686    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl" event={"ID":"8f49f690-e0bd-43d5-bb48-2b45feaf2a4f","Type":"ContainerDied","Data":"61152922c5b29d47ca27dd50124ab59d739d03aa87f4493d515758f8b56c08e3"}
Oct 31 01:31:35.094959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.094744    5419 scope.go:122] "RemoveContainer" containerID="4abdd112adfe55d4ef7b9f82b7462f208a3256e7aff7d3f1a89d4db41124b993"
Oct 31 01:31:35.096781 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.096754    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj" event={"ID":"bb32e360-3c70-43bd-a19a-86d25a2d0d3e","Type":"ContainerDied","Data":"da55b19e58e0aae10091706e1e0c26e76f8d38a8f9d5f49c4d9c10361cd297c8"}
Oct 31 01:31:35.096869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.096855    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"
Oct 31 01:31:35.100770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.100337    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr" event={"ID":"ad5493b4-7690-4d3a-8b69-4f8105e82abc","Type":"ContainerDied","Data":"7460b89ccf3b70f4c3f530e4fff591dcd19f407f27d82e7344df63284c9b459a"}
Oct 31 01:31:35.100770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.100354    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"
Oct 31 01:31:35.101963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.101943    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"
Oct 31 01:31:35.102305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.101943    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w" event={"ID":"b5b6bbbf-5f94-4689-89d5-fb54ff093167","Type":"ContainerDied","Data":"2753df029aba3f1f572956be8354aa419608a96759c487ae64c046f85019bcaf"}
Oct 31 01:31:35.111355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.111328    5419 scope.go:122] "RemoveContainer" containerID="e8c66f3d10ad61f1f3ce807b2f0a2765479aea0f42ac4381dfc4802f63c36c7d"
Oct 31 01:31:35.122237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.122217    5419 scope.go:122] "RemoveContainer" containerID="3ad73901e95a45ec3ddd583bf3b2be84a86023d4c91a590e718a5dbef00e0937"
Oct 31 01:31:35.131566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.131547    5419 scope.go:122] "RemoveContainer" containerID="b5981b42a6b42e68b4302d30b04e9f3ae4ac7ae779491d589981d7383db57365"
Oct 31 01:31:35.259516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.259481    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"]
Oct 31 01:31:35.308809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.308768    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-r5kjj"]
Oct 31 01:31:35.417882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.417810    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"]
Oct 31 01:31:35.423616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.423586    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-8xf9w"]
Oct 31 01:31:35.532187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.532123    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"]
Oct 31 01:31:35.535073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.535044    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-6wstr"]
Oct 31 01:31:35.642008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.641972    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"]
Oct 31 01:31:35.645267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:35.645243    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-lcbbl"]
Oct 31 01:31:36.839068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:36.839041    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" path="/var/lib/kubelet/pods/8f49f690-e0bd-43d5-bb48-2b45feaf2a4f/volumes"
Oct 31 01:31:36.839554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:36.839541    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" path="/var/lib/kubelet/pods/ad5493b4-7690-4d3a-8b69-4f8105e82abc/volumes"
Oct 31 01:31:36.839979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:36.839966    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" path="/var/lib/kubelet/pods/b5b6bbbf-5f94-4689-89d5-fb54ff093167/volumes"
Oct 31 01:31:36.840591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:36.840574    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" path="/var/lib/kubelet/pods/bb32e360-3c70-43bd-a19a-86d25a2d0d3e/volumes"
Oct 31 01:31:38.072259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.072227    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"]
Oct 31 01:31:38.072989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.072970    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy"
Oct 31 01:31:38.072989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.072990    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073002    5419 cpu_manager.go:414] "containerMap: removing container" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073008    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073014    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073019    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073036    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073040    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073048    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy"
Oct 31 01:31:38.073080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073052    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy"
Oct 31 01:31:38.073343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073160    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy"
Oct 31 01:31:38.073343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073168    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb32e360-3c70-43bd-a19a-86d25a2d0d3e" containerName="istio-proxy"
Oct 31 01:31:38.073343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073173    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5493b4-7690-4d3a-8b69-4f8105e82abc" containerName="istio-proxy"
Oct 31 01:31:38.073343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073181    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f49f690-e0bd-43d5-bb48-2b45feaf2a4f" containerName="istio-proxy"
Oct 31 01:31:38.073343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.073187    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b6bbbf-5f94-4689-89d5-fb54ff093167" containerName="istio-proxy"
Oct 31 01:31:38.078405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.078387    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.133863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.133830    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"]
Oct 31 01:31:38.173378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.173333    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.173620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.173601    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swx6\" (UniqueName: \"kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6\") pod \"payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.181641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.181538    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-57ea84\""
Oct 31 01:31:38.182099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.182076    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-57ea84-dockercfg-ttmbn\""
Oct 31 01:31:38.281307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.281269    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.281586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.281567    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swx6\" (UniqueName: \"kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6\") pod \"payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.304191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.282865    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.374226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.363981    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swx6\" (UniqueName: \"kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6\") pod \"payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.411225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.392857    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:38.432916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.432881    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"]
Oct 31 01:31:38.461363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.461330    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"]
Oct 31 01:31:38.461520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.461475    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.595922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.595892    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j284t\" (UniqueName: \"kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t\") pod \"payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") " pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.596101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.595942    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") " pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.697792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.697708    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j284t\" (UniqueName: \"kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t\") pod \"payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") " pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.697792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.697778    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") " pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.698521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.698502    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") " pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.754100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.754064    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j284t\" (UniqueName: \"kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t\") pod \"payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") " pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.795242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.795213    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:38.838376 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:38.838335    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f51ada_f315_43cd_a9cc_7ca244b5c64d.slice/crio-8e91c13c00dba1c9e266638fb8cde7fcd629138c45d8404bb6a197e72783d944 WatchSource:0}: Error finding container 8e91c13c00dba1c9e266638fb8cde7fcd629138c45d8404bb6a197e72783d944: Status 404 returned error can't find the container with id 8e91c13c00dba1c9e266638fb8cde7fcd629138c45d8404bb6a197e72783d944
Oct 31 01:31:38.840789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:38.840763    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"]
Oct 31 01:31:39.136900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.136860    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg" event={"ID":"96f51ada-f315-43cd-a9cc-7ca244b5c64d","Type":"ContainerStarted","Data":"d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849"}
Oct 31 01:31:39.137383 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.136924    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg" event={"ID":"96f51ada-f315-43cd-a9cc-7ca244b5c64d","Type":"ContainerStarted","Data":"8e91c13c00dba1c9e266638fb8cde7fcd629138c45d8404bb6a197e72783d944"}
Oct 31 01:31:39.137383 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.136952    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:39.174262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.174233    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"]
Oct 31 01:31:39.175183 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:39.175128    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a978b4e_8e87_40e8_9e0e_28f014868bb5.slice/crio-fdf52b447c0785b93a20337d896456896d43242dd99daf56e6b5fe59bf96de90 WatchSource:0}: Error finding container fdf52b447c0785b93a20337d896456896d43242dd99daf56e6b5fe59bf96de90: Status 404 returned error can't find the container with id fdf52b447c0785b93a20337d896456896d43242dd99daf56e6b5fe59bf96de90
Oct 31 01:31:39.251622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.251574    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg" podStartSLOduration=1.251560187 podStartE2EDuration="1.251560187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:39.248600297 +0000 UTC m=+1620.549556874" watchObservedRunningTime="2026-10-31 01:31:39.251560187 +0000 UTC m=+1620.552516793"
Oct 31 01:31:39.774288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.770575    5419 scope.go:122] "RemoveContainer" containerID="b2c27dced0b558e7a59590589b44b0b9d6706c6577a8b952f45eab23a53063a2"
Oct 31 01:31:39.802610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.799384    5419 scope.go:122] "RemoveContainer" containerID="a6a674a84e9ba96ca142490918a6f2f03e3fcad8a7a46876425cbaf3435ac40f"
Oct 31 01:31:39.816436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.816415    5419 scope.go:122] "RemoveContainer" containerID="ee958fab662d298b4b5c6ffe8500aa4a95f22ac025319e1508b12a31d143397d"
Oct 31 01:31:39.849661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.849643    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"]
Oct 31 01:31:39.855031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.855015    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:39.907499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.907465    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:31:39.907499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.907465    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:31:39.907749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.907594    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:31:39.909056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.909031    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:31:39.910502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.910483    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"]
Oct 31 01:31:39.933569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.933523    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:39.933764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.933743    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:39.933966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:39.933942    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlp2f\" (UniqueName: \"kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.035111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.035037    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.035276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.035177    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlp2f\" (UniqueName: \"kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.035320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.035283    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.035919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.035898    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.037592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.037570    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.091647 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.091616    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlp2f\" (UniqueName: \"kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f\") pod \"maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.144618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.144575    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj" event={"ID":"4a978b4e-8e87-40e8-9e0e-28f014868bb5","Type":"ContainerStarted","Data":"3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d"}
Oct 31 01:31:40.145077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.144643    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj" event={"ID":"4a978b4e-8e87-40e8-9e0e-28f014868bb5","Type":"ContainerStarted","Data":"fdf52b447c0785b93a20337d896456896d43242dd99daf56e6b5fe59bf96de90"}
Oct 31 01:31:40.145077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.144779    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:31:40.171179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.171160    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:40.262047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.261982    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj" podStartSLOduration=2.261961882 podStartE2EDuration="2.261961882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:40.256890979 +0000 UTC m=+1621.557847556" watchObservedRunningTime="2026-10-31 01:31:40.261961882 +0000 UTC m=+1621.562918459"
Oct 31 01:31:40.569237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:40.569211    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"]
Oct 31 01:31:40.570341 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:31:40.570304    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0219e1f5_e61c_444a_8236_ab920337f0de.slice/crio-b799938188dc0380e05b86302b7193fc190073a0b6c473cbefebdd93c70aedc7 WatchSource:0}: Error finding container b799938188dc0380e05b86302b7193fc190073a0b6c473cbefebdd93c70aedc7: Status 404 returned error can't find the container with id b799938188dc0380e05b86302b7193fc190073a0b6c473cbefebdd93c70aedc7
Oct 31 01:31:41.153444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:41.153355    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp" event={"ID":"0219e1f5-e61c-444a-8236-ab920337f0de","Type":"ContainerStarted","Data":"710f8fcc14f550e8703e6902c20a1a19a9fef83bfc4a4cdcd17dffd9b2543df5"}
Oct 31 01:31:41.153444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:41.153413    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp" event={"ID":"0219e1f5-e61c-444a-8236-ab920337f0de","Type":"ContainerStarted","Data":"b799938188dc0380e05b86302b7193fc190073a0b6c473cbefebdd93c70aedc7"}
Oct 31 01:31:41.267113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:41.267051    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp" podStartSLOduration=2.267038966 podStartE2EDuration="2.267038966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:31:41.265574534 +0000 UTC m=+1622.566531113" watchObservedRunningTime="2026-10-31 01:31:41.267038966 +0000 UTC m=+1622.567995542"
Oct 31 01:31:42.165412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:42.165382    5419 generic.go:356] "Generic (PLEG): container finished" podID="0219e1f5-e61c-444a-8236-ab920337f0de" containerID="710f8fcc14f550e8703e6902c20a1a19a9fef83bfc4a4cdcd17dffd9b2543df5" exitCode=0
Oct 31 01:31:42.165812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:42.165470    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp" event={"ID":"0219e1f5-e61c-444a-8236-ab920337f0de","Type":"ContainerDied","Data":"710f8fcc14f550e8703e6902c20a1a19a9fef83bfc4a4cdcd17dffd9b2543df5"}
Oct 31 01:31:43.044856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.044824    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"]
Oct 31 01:31:43.094048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.094017    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4ba8ff-tf7g2"]
Oct 31 01:31:43.328573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.328549    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:43.496265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.496233    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca\") pod \"0219e1f5-e61c-444a-8236-ab920337f0de\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") "
Oct 31 01:31:43.496437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.496343    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca\") pod \"0219e1f5-e61c-444a-8236-ab920337f0de\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") "
Oct 31 01:31:43.496551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.496531    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f\" (UniqueName: \"kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f\") pod \"0219e1f5-e61c-444a-8236-ab920337f0de\" (UID: \"0219e1f5-e61c-444a-8236-ab920337f0de\") "
Oct 31 01:31:43.496784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.496754    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca" pod "0219e1f5-e61c-444a-8236-ab920337f0de" (UID: "0219e1f5-e61c-444a-8236-ab920337f0de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:43.497158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.497118    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:43.498493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.498475    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca" pod "0219e1f5-e61c-444a-8236-ab920337f0de" (UID: "0219e1f5-e61c-444a-8236-ab920337f0de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:31:43.498830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.498800    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f" pod "0219e1f5-e61c-444a-8236-ab920337f0de" (UID: "0219e1f5-e61c-444a-8236-ab920337f0de"). InnerVolumeSpecName "kube-api-access-tlp2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:43.598573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.598498    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0219e1f5-e61c-444a-8236-ab920337f0de-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:43.598573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:43.598525    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlp2f\" (UniqueName: \"kubernetes.io/projected/0219e1f5-e61c-444a-8236-ab920337f0de-kube-api-access-tlp2f\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:31:44.180015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:44.179982    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"
Oct 31 01:31:44.180015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:44.180006    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp" event={"ID":"0219e1f5-e61c-444a-8236-ab920337f0de","Type":"ContainerDied","Data":"b799938188dc0380e05b86302b7193fc190073a0b6c473cbefebdd93c70aedc7"}
Oct 31 01:31:44.180322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:44.180063    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b799938188dc0380e05b86302b7193fc190073a0b6c473cbefebdd93c70aedc7"
Oct 31 01:31:44.583382 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:31:44.583336    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:31:44.839693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:44.839612    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59eb89eb-96df-42b7-ab07-0410e9c74c6d" path="/var/lib/kubelet/pods/59eb89eb-96df-42b7-ab07-0410e9c74c6d/volumes"
Oct 31 01:31:50.149511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:50.149421    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:31:51.157098 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:31:51.157071    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:32:12.424938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.424908    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"]
Oct 31 01:32:12.425578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.425562    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0219e1f5-e61c-444a-8236-ab920337f0de" containerName="revoke-keys"
Oct 31 01:32:12.425626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.425579    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0219e1f5-e61c-444a-8236-ab920337f0de" containerName="revoke-keys"
Oct 31 01:32:12.425679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.425670    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0219e1f5-e61c-444a-8236-ab920337f0de" containerName="revoke-keys"
Oct 31 01:32:12.430524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.430508    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.472595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.472556    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.472745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.472620    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkh2\" (UniqueName: \"kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.472853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.472831    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.485460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.485434    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"]
Oct 31 01:32:12.496670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.496648    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:32:12.496670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.496647    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:32:12.530377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.530351    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:32:12.530377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.530350    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:32:12.574412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.574381    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.574553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.574458    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.574553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.574497    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkh2\" (UniqueName: \"kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.575030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.574994    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.576892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.576876    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.631030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.631004    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkh2\" (UniqueName: \"kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2\") pod \"maas-api-revoke-sub-294520ea-nsmkl\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:12.744678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:12.744648    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:13.122188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:13.122159    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"]
Oct 31 01:32:13.122741 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:32:13.122714    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbf8154_0759_4f2a_b8ca_f2085452d83a.slice/crio-81d68df6494006cffbf46fb5efcba282eefec125defb0dfbc58f01d9308a25d1 WatchSource:0}: Error finding container 81d68df6494006cffbf46fb5efcba282eefec125defb0dfbc58f01d9308a25d1: Status 404 returned error can't find the container with id 81d68df6494006cffbf46fb5efcba282eefec125defb0dfbc58f01d9308a25d1
Oct 31 01:32:13.392114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:13.392085    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl" event={"ID":"dcbf8154-0759-4f2a-b8ca-f2085452d83a","Type":"ContainerStarted","Data":"81d68df6494006cffbf46fb5efcba282eefec125defb0dfbc58f01d9308a25d1"}
Oct 31 01:32:14.400607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:14.400515    5419 generic.go:356] "Generic (PLEG): container finished" podID="dcbf8154-0759-4f2a-b8ca-f2085452d83a" containerID="b2ef38b9681964f24af375e7300875b945f891c6a231c09eaef7a47d1e952f9e" exitCode=0
Oct 31 01:32:14.401026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:14.400601    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl" event={"ID":"dcbf8154-0759-4f2a-b8ca-f2085452d83a","Type":"ContainerDied","Data":"b2ef38b9681964f24af375e7300875b945f891c6a231c09eaef7a47d1e952f9e"}
Oct 31 01:32:14.582736 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:32:14.582699    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:32:15.569484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.569457    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:15.643289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.643259    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca\") pod \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") "
Oct 31 01:32:15.643458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.643318    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2\" (UniqueName: \"kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2\") pod \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") "
Oct 31 01:32:15.643458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.643433    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca\") pod \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\" (UID: \"dcbf8154-0759-4f2a-b8ca-f2085452d83a\") "
Oct 31 01:32:15.643789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.643764    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca" pod "dcbf8154-0759-4f2a-b8ca-f2085452d83a" (UID: "dcbf8154-0759-4f2a-b8ca-f2085452d83a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:15.644034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.644016    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:15.645584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.645558    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca" pod "dcbf8154-0759-4f2a-b8ca-f2085452d83a" (UID: "dcbf8154-0759-4f2a-b8ca-f2085452d83a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:15.645584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.645560    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2" pod "dcbf8154-0759-4f2a-b8ca-f2085452d83a" (UID: "dcbf8154-0759-4f2a-b8ca-f2085452d83a"). InnerVolumeSpecName "kube-api-access-qfkh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:15.744951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.744915    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbf8154-0759-4f2a-b8ca-f2085452d83a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:15.744951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:15.744945    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkh2\" (UniqueName: \"kubernetes.io/projected/dcbf8154-0759-4f2a-b8ca-f2085452d83a-kube-api-access-qfkh2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:16.338165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:16.332062    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:32:16.415838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:16.415806    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl" event={"ID":"dcbf8154-0759-4f2a-b8ca-f2085452d83a","Type":"ContainerDied","Data":"81d68df6494006cffbf46fb5efcba282eefec125defb0dfbc58f01d9308a25d1"}
Oct 31 01:32:16.416014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:16.415858    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d68df6494006cffbf46fb5efcba282eefec125defb0dfbc58f01d9308a25d1"
Oct 31 01:32:16.416014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:16.415913    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"
Oct 31 01:32:24.044359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:24.044319    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"]
Oct 31 01:32:24.093737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:24.093698    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ccf031-7rgqs"]
Oct 31 01:32:24.839344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:24.839306    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c651d34c-8f2a-43fb-b7e7-0dcca9827531" path="/var/lib/kubelet/pods/c651d34c-8f2a-43fb-b7e7-0dcca9827531/volumes"
Oct 31 01:32:26.047217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:26.047177    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"]
Oct 31 01:32:26.096860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:26.096825    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ac23bbc-p2qrp"]
Oct 31 01:32:26.838358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:26.838270    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb32303-d237-442c-89bb-e0c5584615f0" path="/var/lib/kubelet/pods/1cb32303-d237-442c-89bb-e0c5584615f0/volumes"
Oct 31 01:32:27.493637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:27.493605    5419 generic.go:356] "Generic (PLEG): container finished" podID="e4228b72-6d15-46f9-a069-c68007a190a9" containerID="47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f" exitCode=0
Oct 31 01:32:27.494033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:27.493688    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" event={"ID":"e4228b72-6d15-46f9-a069-c68007a190a9","Type":"ContainerDied","Data":"47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"}
Oct 31 01:32:27.494211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:27.494194    5419 scope.go:122] "RemoveContainer" containerID="47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"
Oct 31 01:32:27.495695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:27.495670    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:32:27.495769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:27.495736    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:32:27.495817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:27.495769    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:32:28.529160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:28.527492    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" event={"ID":"e4228b72-6d15-46f9-a069-c68007a190a9","Type":"ContainerStarted","Data":"a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e"}
Oct 31 01:32:28.711494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:28.711463    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:32:28.721864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:28.721833    5419 patch_prober.go:29] interesting pod/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:32:28.722152 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:28.722098    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 01:32:29.710418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:29.710377    5419 patch_prober.go:29] interesting pod/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:32:29.710796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:29.710445    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 01:32:30.710185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:30.710121    5419 patch_prober.go:29] interesting pod/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:32:30.710374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:30.710209    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 01:32:31.710088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:31.710051    5419 patch_prober.go:29] interesting pod/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:32:31.710606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:31.710110    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 01:32:32.710755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:32.710687    5419 patch_prober.go:29] interesting pod/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 01:32:32.710755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:32.710740    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 01:32:33.103189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:33.103150    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:32:33.797524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:33.797489    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:32:33.806263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:33.806235    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:32:35.630618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:35.630581    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"]
Oct 31 01:32:35.679935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:35.679900    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7cec02-sfpxk"]
Oct 31 01:32:36.838340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:36.838309    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b23326a5-fbea-49b6-81cf-28d29848cd16" path="/var/lib/kubelet/pods/b23326a5-fbea-49b6-81cf-28d29848cd16/volumes"
Oct 31 01:32:37.046131 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:37.046101    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"]
Oct 31 01:32:37.095345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:37.095264    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecc7c35d-5ztdz"]
Oct 31 01:32:38.837931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:38.837893    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a09efb7-fa0e-4b24-936a-86adaebd9088" path="/var/lib/kubelet/pods/1a09efb7-fa0e-4b24-936a-86adaebd9088/volumes"
Oct 31 01:32:39.877001 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:39.876966    5419 scope.go:122] "RemoveContainer" containerID="2214573a464ff4ec8a311c56d72b8b7dc54b6f05721996cbe7eb853d3e194f99"
Oct 31 01:32:43.033378 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.033347    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"]
Oct 31 01:32:43.034016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.034000    5419 cpu_manager.go:414] "containerMap: removing container" podUID="dcbf8154-0759-4f2a-b8ca-f2085452d83a" containerName="revoke-keys"
Oct 31 01:32:43.034063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.034018    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbf8154-0759-4f2a-b8ca-f2085452d83a" containerName="revoke-keys"
Oct 31 01:32:43.034116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.034107    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbf8154-0759-4f2a-b8ca-f2085452d83a" containerName="revoke-keys"
Oct 31 01:32:43.039026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.039010    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.091717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.091691    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:32:43.136658 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.136626    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlvsp\" (UniqueName: \"kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.136858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.136672    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.136858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.136755    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.141173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.141130    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:32:43.141293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.141264    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:32:43.141351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.141292    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:32:43.144544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.144526    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"]
Oct 31 01:32:43.237708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.237677    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlvsp\" (UniqueName: \"kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.237916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.237724    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.237916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.237764    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.238415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.238388    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.240297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.240275    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.293881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.293824    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlvsp\" (UniqueName: \"kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp\") pod \"maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.352653 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.352627    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:43.747665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:43.747628    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"]
Oct 31 01:32:43.748501 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:32:43.748473    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2a73ad_d213_4aea_9dad_9a5923b4fd9e.slice/crio-59e50bdcbd968ad3b4d886df514e38639cd64032c0e4b16e37b5d1efe3e034ed WatchSource:0}: Error finding container 59e50bdcbd968ad3b4d886df514e38639cd64032c0e4b16e37b5d1efe3e034ed: Status 404 returned error can't find the container with id 59e50bdcbd968ad3b4d886df514e38639cd64032c0e4b16e37b5d1efe3e034ed
Oct 31 01:32:44.471742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:44.471712    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:32:44.584280 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:32:44.584242    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:32:44.710457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:44.710419    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p" event={"ID":"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e","Type":"ContainerStarted","Data":"8db1e08c900f60e2fee63489047901b36206d1a7f9bb615771d9f7a963cd9c94"}
Oct 31 01:32:44.710620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:44.710482    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p" event={"ID":"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e","Type":"ContainerStarted","Data":"59e50bdcbd968ad3b4d886df514e38639cd64032c0e4b16e37b5d1efe3e034ed"}
Oct 31 01:32:44.827471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:44.827422    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p" podStartSLOduration=2.827404425 podStartE2EDuration="2.827404425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:44.823754036 +0000 UTC m=+1686.124710614" watchObservedRunningTime="2026-10-31 01:32:44.827404425 +0000 UTC m=+1686.128361001"
Oct 31 01:32:45.774170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:45.771204    5419 generic.go:356] "Generic (PLEG): container finished" podID="7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" containerID="8db1e08c900f60e2fee63489047901b36206d1a7f9bb615771d9f7a963cd9c94" exitCode=0
Oct 31 01:32:45.774170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:45.771333    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p" event={"ID":"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e","Type":"ContainerDied","Data":"8db1e08c900f60e2fee63489047901b36206d1a7f9bb615771d9f7a963cd9c94"}
Oct 31 01:32:46.615166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:46.602256    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:32:47.752036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:47.751993    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": context deadline exceeded"
Oct 31 01:32:47.840996 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:47.833292    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:32:47.887426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:47.887381    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:48.036602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.036522    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca\") pod \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") "
Oct 31 01:32:48.036768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.036628    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp\" (UniqueName: \"kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp\") pod \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") "
Oct 31 01:32:48.036815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.036770    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca\") pod \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\" (UID: \"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e\") "
Oct 31 01:32:48.037211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.037180    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca" pod "7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" (UID: "7d2a73ad-d213-4aea-9dad-9a5923b4fd9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:48.037507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.037446    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:48.038769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.038745    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca" pod "7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" (UID: "7d2a73ad-d213-4aea-9dad-9a5923b4fd9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:48.038910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.038894    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp" pod "7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" (UID: "7d2a73ad-d213-4aea-9dad-9a5923b4fd9e"). InnerVolumeSpecName "kube-api-access-tlvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:48.138581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.138550    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:48.138581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.138580    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlvsp\" (UniqueName: \"kubernetes.io/projected/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e-kube-api-access-tlvsp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:48.856339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.856314    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"
Oct 31 01:32:48.856711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.856346    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p" event={"ID":"7d2a73ad-d213-4aea-9dad-9a5923b4fd9e","Type":"ContainerDied","Data":"59e50bdcbd968ad3b4d886df514e38639cd64032c0e4b16e37b5d1efe3e034ed"}
Oct 31 01:32:48.856711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:48.856396    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e50bdcbd968ad3b4d886df514e38639cd64032c0e4b16e37b5d1efe3e034ed"
Oct 31 01:32:53.622003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.621969    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"]
Oct 31 01:32:53.622727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.622711    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" containerName="revoke-keys"
Oct 31 01:32:53.622784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.622729    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" containerName="revoke-keys"
Oct 31 01:32:53.622896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.622885    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" containerName="revoke-keys"
Oct 31 01:32:53.626857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.626841    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.662249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.662210    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.662446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.662345    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.662446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.662414    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6npq\" (UniqueName: \"kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.684467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.684431    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"]
Oct 31 01:32:53.729284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.729249    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:32:53.729284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.729248    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:32:53.729593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.729572    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:32:53.729663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.729572    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:32:53.763648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.763601    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.763834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.763712    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.763834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.763754    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6npq\" (UniqueName: \"kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.764504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.764479    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.766321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.766300    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.821415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.821380    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6npq\" (UniqueName: \"kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq\") pod \"maas-api-revoke-sub-da23bad0-fdmwp\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:53.943109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:53.943022    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:54.338795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:54.338762    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"]
Oct 31 01:32:54.340921 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:32:54.340897    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9f33fe_54c3_4326_9621_7c0eddf586c5.slice/crio-89617bc30c28be728d6b968a1b296d591bccedf91a27e720b3cafa59053b9c83 WatchSource:0}: Error finding container 89617bc30c28be728d6b968a1b296d591bccedf91a27e720b3cafa59053b9c83: Status 404 returned error can't find the container with id 89617bc30c28be728d6b968a1b296d591bccedf91a27e720b3cafa59053b9c83
Oct 31 01:32:55.171626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:55.171574    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp" event={"ID":"1e9f33fe-54c3-4326-9621-7c0eddf586c5","Type":"ContainerStarted","Data":"04f718dacfe8f70712b7328a83c3c0b7eef16eb1a56e2fbff784385c3966f059"}
Oct 31 01:32:55.172290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:55.172267    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp" event={"ID":"1e9f33fe-54c3-4326-9621-7c0eddf586c5","Type":"ContainerStarted","Data":"89617bc30c28be728d6b968a1b296d591bccedf91a27e720b3cafa59053b9c83"}
Oct 31 01:32:55.287059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:55.287007    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp" podStartSLOduration=2.286992871 podStartE2EDuration="2.286992871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:55.283990977 +0000 UTC m=+1696.584947558" watchObservedRunningTime="2026-10-31 01:32:55.286992871 +0000 UTC m=+1696.587949442"
Oct 31 01:32:56.207162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:56.203953    5419 generic.go:356] "Generic (PLEG): container finished" podID="1e9f33fe-54c3-4326-9621-7c0eddf586c5" containerID="04f718dacfe8f70712b7328a83c3c0b7eef16eb1a56e2fbff784385c3966f059" exitCode=0
Oct 31 01:32:56.207162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:56.204249    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp" event={"ID":"1e9f33fe-54c3-4326-9621-7c0eddf586c5","Type":"ContainerDied","Data":"04f718dacfe8f70712b7328a83c3c0b7eef16eb1a56e2fbff784385c3966f059"}
Oct 31 01:32:57.212442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.212356    5419 generic.go:356] "Generic (PLEG): container finished" podID="1f200f95-279c-4bde-8336-d3975396e88b" containerID="d5ec4f1e5cbe88e7e69ba5a8505ef85d81a139b89067443ae0c847a6f34bfc3e" exitCode=0
Oct 31 01:32:57.212961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.212434    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" event={"ID":"1f200f95-279c-4bde-8336-d3975396e88b","Type":"ContainerDied","Data":"d5ec4f1e5cbe88e7e69ba5a8505ef85d81a139b89067443ae0c847a6f34bfc3e"}
Oct 31 01:32:57.213497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.213038    5419 scope.go:122] "RemoveContainer" containerID="d5ec4f1e5cbe88e7e69ba5a8505ef85d81a139b89067443ae0c847a6f34bfc3e"
Oct 31 01:32:57.215064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.215036    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:32:57.215158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.215107    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:32:57.215204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.215155    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:32:57.394101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.394071    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:57.445637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.445601    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq\" (UniqueName: \"kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq\") pod \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") "
Oct 31 01:32:57.445800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.445735    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca\") pod \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") "
Oct 31 01:32:57.445898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.445881    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca\") pod \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\" (UID: \"1e9f33fe-54c3-4326-9621-7c0eddf586c5\") "
Oct 31 01:32:57.446621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.446597    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca" pod "1e9f33fe-54c3-4326-9621-7c0eddf586c5" (UID: "1e9f33fe-54c3-4326-9621-7c0eddf586c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:32:57.453288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.453262    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca" pod "1e9f33fe-54c3-4326-9621-7c0eddf586c5" (UID: "1e9f33fe-54c3-4326-9621-7c0eddf586c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:32:57.453405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.453291    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq" pod "1e9f33fe-54c3-4326-9621-7c0eddf586c5" (UID: "1e9f33fe-54c3-4326-9621-7c0eddf586c5"). InnerVolumeSpecName "kube-api-access-x6npq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:32:57.549162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.548482    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6npq\" (UniqueName: \"kubernetes.io/projected/1e9f33fe-54c3-4326-9621-7c0eddf586c5-kube-api-access-x6npq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:57.549162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.548521    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:57.549162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:57.548536    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9f33fe-54c3-4326-9621-7c0eddf586c5-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:32:58.220426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.220394    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"
Oct 31 01:32:58.220846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.220434    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp" event={"ID":"1e9f33fe-54c3-4326-9621-7c0eddf586c5","Type":"ContainerDied","Data":"89617bc30c28be728d6b968a1b296d591bccedf91a27e720b3cafa59053b9c83"}
Oct 31 01:32:58.220846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.220487    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89617bc30c28be728d6b968a1b296d591bccedf91a27e720b3cafa59053b9c83"
Oct 31 01:32:58.222636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.222611    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" event={"ID":"1f200f95-279c-4bde-8336-d3975396e88b","Type":"ContainerStarted","Data":"f3c9916f17e7fcfbc0f9980d947c02ff9bf5e442648a477069386577801a8b03"}
Oct 31 01:32:58.299347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.299312    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:32:58.300622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.300597    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:32:58.300761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:58.300645    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 31 01:32:59.297175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:59.296536    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:32:59.297175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:32:59.296598    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 31 01:33:00.287574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:00.287535    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:33:00.287880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:00.287847    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 31 01:33:01.299176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:01.298257    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:33:01.299176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:01.298321    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 31 01:33:01.990164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:01.985237    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:33:02.299635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:02.299530    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:33:02.299635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:02.299586    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 31 01:33:03.277376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:03.277340    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Oct 31 01:33:03.277589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:03.277411    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Oct 31 01:33:04.281364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:04.281334    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:33:04.282547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:04.282524    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:33:06.249737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:06.249706    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"]
Oct 31 01:33:06.250244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:06.249910    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy" containerID="cri-o://696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0" gracePeriod=30
Oct 31 01:33:06.299491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:06.299461    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"]
Oct 31 01:33:06.299783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:06.299757    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy" containerID="cri-o://c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19" gracePeriod=30
Oct 31 01:33:09.132271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.132238    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"]
Oct 31 01:33:09.132836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.132822    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1e9f33fe-54c3-4326-9621-7c0eddf586c5" containerName="revoke-keys"
Oct 31 01:33:09.132881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.132837    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9f33fe-54c3-4326-9621-7c0eddf586c5" containerName="revoke-keys"
Oct 31 01:33:09.132964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.132955    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9f33fe-54c3-4326-9621-7c0eddf586c5" containerName="revoke-keys"
Oct 31 01:33:09.136530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.136512    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.192252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.192212    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.192423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.192399    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcfv\" (UniqueName: \"kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.192507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.192466    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.193481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.193456    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"]
Oct 31 01:33:09.238506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.238478    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:33:09.238724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.238712    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:33:09.238794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.238760    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:33:09.238861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.238844    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:33:09.293482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.293448    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcfv\" (UniqueName: \"kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.293652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.293507    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.293652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.293607    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.294129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.294107    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.296106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.296085    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.349943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.349918    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcfv\" (UniqueName: \"kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv\") pod \"maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.450200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.450114    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:09.860856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:09.860811    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"]
Oct 31 01:33:09.896679 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:33:09.896646    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4d9b96_cc50_4358_8571_174d1d6cef11.slice/crio-25e99955e564cb97d8e51eb4b3e4ea53ef1f286150b731da29155494bae0d180 WatchSource:0}: Error finding container 25e99955e564cb97d8e51eb4b3e4ea53ef1f286150b731da29155494bae0d180: Status 404 returned error can't find the container with id 25e99955e564cb97d8e51eb4b3e4ea53ef1f286150b731da29155494bae0d180
Oct 31 01:33:10.338810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:10.336942    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr" event={"ID":"0f4d9b96-cc50-4358-8571-174d1d6cef11","Type":"ContainerStarted","Data":"cc4143bc92aac7111a0a2f34ef530137a69327f8f654bab379f8ac78052193cc"}
Oct 31 01:33:10.338810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:10.337016    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr" event={"ID":"0f4d9b96-cc50-4358-8571-174d1d6cef11","Type":"ContainerStarted","Data":"25e99955e564cb97d8e51eb4b3e4ea53ef1f286150b731da29155494bae0d180"}
Oct 31 01:33:10.450471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:10.450367    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr" podStartSLOduration=1.450353535 podStartE2EDuration="1.450353535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:10.449278331 +0000 UTC m=+1711.750234923" watchObservedRunningTime="2026-10-31 01:33:10.450353535 +0000 UTC m=+1711.751310112"
Oct 31 01:33:11.345349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.345315    5419 generic.go:356] "Generic (PLEG): container finished" podID="0f4d9b96-cc50-4358-8571-174d1d6cef11" containerID="cc4143bc92aac7111a0a2f34ef530137a69327f8f654bab379f8ac78052193cc" exitCode=0
Oct 31 01:33:11.345791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.345404    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr" event={"ID":"0f4d9b96-cc50-4358-8571-174d1d6cef11","Type":"ContainerDied","Data":"cc4143bc92aac7111a0a2f34ef530137a69327f8f654bab379f8ac78052193cc"}
Oct 31 01:33:11.579265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.579237    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:33:11.688663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.688618    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.688992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.688853    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.688992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.688983    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689030    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689085    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.689233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689113    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689237    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689323    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689357    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689387    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh\") pod \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\" (UID: \"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71\") "
Oct 31 01:33:11.689716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.689695    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.690686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.690127    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.690686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.690170    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.690686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.690370    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.690686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.690507    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.691120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.691090    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:11.695605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.695579    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.695703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.695672    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:11.730428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.730384    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "kube-api-access-974fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:11.732713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.732662    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:33:11.734738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.734699    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo" pod "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" (UID: "11ea7f89-4b8c-41d4-8107-f95ad9aa6f71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:11.782989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.782954    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"]
Oct 31 01:33:11.785767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.785740    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01cbd8-h85vp"]
Oct 31 01:33:11.790889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.790865    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.790907    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791093    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791207    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.791240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791221    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791263    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791337    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791348    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.791397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791379    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791448    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.791563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791452    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791516    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket\") pod \"a44131ca-439b-431e-86b0-88edb2b5e9da\" (UID: \"a44131ca-439b-431e-86b0-88edb2b5e9da\") "
Oct 31 01:33:11.791822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791793    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:11.791989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.791969    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.792255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792223    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792242    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792257    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792270    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792283    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792291    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a44131ca-439b-431e-86b0-88edb2b5e9da-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792310    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792321    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792330    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792339    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792348    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974fh\" (UniqueName: \"kubernetes.io/projected/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71-kube-api-access-974fh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.792389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.792356    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.793302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.793281    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "kube-api-access-vmr7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:11.793720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.793707    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:11.794210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.794189    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:11.794305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.794224    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token" pod "a44131ca-439b-431e-86b0-88edb2b5e9da" (UID: "a44131ca-439b-431e-86b0-88edb2b5e9da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:11.893360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.893268    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.893360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.893299    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.893360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.893313    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:11.893360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:11.893325    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmr7q\" (UniqueName: \"kubernetes.io/projected/a44131ca-439b-431e-86b0-88edb2b5e9da-kube-api-access-vmr7q\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:12.353599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.353563    5419 generic.go:356] "Generic (PLEG): container finished" podID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerID="696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0" exitCode=0
Oct 31 01:33:12.353931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.353673    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"
Oct 31 01:33:12.353931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.353703    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" event={"ID":"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71","Type":"ContainerDied","Data":"696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0"}
Oct 31 01:33:12.353931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.353760    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq" event={"ID":"11ea7f89-4b8c-41d4-8107-f95ad9aa6f71","Type":"ContainerDied","Data":"83c183032a2c120cfd36b0bf80a076772152463ed5c573ec9c8680f449728be2"}
Oct 31 01:33:12.353931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.353783    5419 scope.go:122] "RemoveContainer" containerID="696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0"
Oct 31 01:33:12.355598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.355534    5419 generic.go:356] "Generic (PLEG): container finished" podID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerID="c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19" exitCode=0
Oct 31 01:33:12.355598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.355589    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" event={"ID":"a44131ca-439b-431e-86b0-88edb2b5e9da","Type":"ContainerDied","Data":"c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19"}
Oct 31 01:33:12.355598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.355602    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"
Oct 31 01:33:12.355845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.355640    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9" event={"ID":"a44131ca-439b-431e-86b0-88edb2b5e9da","Type":"ContainerDied","Data":"8f6a45c1d586734cbd4a14054926b9bd3f5f883d1345c4046a92aeeae714359b"}
Oct 31 01:33:12.366242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.366220    5419 scope.go:122] "RemoveContainer" containerID="696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0"
Oct 31 01:33:12.366558 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:12.366532    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0\": container with ID starting with 696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0 not found: ID does not exist" containerID="696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0"
Oct 31 01:33:12.366646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.366562    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0"} err="failed to get container status \"696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0\": rpc error: code = NotFound desc = could not find container \"696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0\": container with ID starting with 696a7c27b299cfd90d859859ea67dbeae70ef6bd4c54d2b411789bc5aab7b8b0 not found: ID does not exist"
Oct 31 01:33:12.366646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.366579    5419 scope.go:122] "RemoveContainer" containerID="c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19"
Oct 31 01:33:12.376770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.376753    5419 scope.go:122] "RemoveContainer" containerID="c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19"
Oct 31 01:33:12.377110 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:12.377085    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19\": container with ID starting with c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19 not found: ID does not exist" containerID="c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19"
Oct 31 01:33:12.377313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.377116    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19"} err="failed to get container status \"c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19\": rpc error: code = NotFound desc = could not find container \"c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19\": container with ID starting with c541d2a31af93466945092cefcc22f5e603bb2b991cbedca8bbe3a7d866cec19 not found: ID does not exist"
Oct 31 01:33:12.514109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.514083    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:12.521527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.521507    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"]
Oct 31 01:33:12.524411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.524394    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-pshh9"]
Oct 31 01:33:12.603682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.603585    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca\") pod \"0f4d9b96-cc50-4358-8571-174d1d6cef11\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") "
Oct 31 01:33:12.603861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.603684    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca\") pod \"0f4d9b96-cc50-4358-8571-174d1d6cef11\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") "
Oct 31 01:33:12.603861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.603800    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv\" (UniqueName: \"kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv\") pod \"0f4d9b96-cc50-4358-8571-174d1d6cef11\" (UID: \"0f4d9b96-cc50-4358-8571-174d1d6cef11\") "
Oct 31 01:33:12.604028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.604004    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca" pod "0f4d9b96-cc50-4358-8571-174d1d6cef11" (UID: "0f4d9b96-cc50-4358-8571-174d1d6cef11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:12.604366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.604343    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:12.605824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.605796    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca" pod "0f4d9b96-cc50-4358-8571-174d1d6cef11" (UID: "0f4d9b96-cc50-4358-8571-174d1d6cef11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:12.606037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.606015    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv" pod "0f4d9b96-cc50-4358-8571-174d1d6cef11" (UID: "0f4d9b96-cc50-4358-8571-174d1d6cef11"). InnerVolumeSpecName "kube-api-access-4bcfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:12.643187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.643158    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"]
Oct 31 01:33:12.665721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.665692    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-br7bq"]
Oct 31 01:33:12.705334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.705303    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4d9b96-cc50-4358-8571-174d1d6cef11-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:12.705334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.705330    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bcfv\" (UniqueName: \"kubernetes.io/projected/0f4d9b96-cc50-4358-8571-174d1d6cef11-kube-api-access-4bcfv\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:12.837762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.837733    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0219e1f5-e61c-444a-8236-ab920337f0de" path="/var/lib/kubelet/pods/0219e1f5-e61c-444a-8236-ab920337f0de/volumes"
Oct 31 01:33:12.838262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.838243    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" path="/var/lib/kubelet/pods/11ea7f89-4b8c-41d4-8107-f95ad9aa6f71/volumes"
Oct 31 01:33:12.838714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:12.838702    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" path="/var/lib/kubelet/pods/a44131ca-439b-431e-86b0-88edb2b5e9da/volumes"
Oct 31 01:33:13.364597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:13.364568    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"
Oct 31 01:33:13.364597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:13.364590    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr" event={"ID":"0f4d9b96-cc50-4358-8571-174d1d6cef11","Type":"ContainerDied","Data":"25e99955e564cb97d8e51eb4b3e4ea53ef1f286150b731da29155494bae0d180"}
Oct 31 01:33:13.365125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:13.364635    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e99955e564cb97d8e51eb4b3e4ea53ef1f286150b731da29155494bae0d180"
Oct 31 01:33:14.584610 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:14.584571    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:33:18.538112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:18.537919    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"]
Oct 31 01:33:18.538603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:18.538166    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy" containerID="cri-o://c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0" gracePeriod=30
Oct 31 01:33:19.386924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.386888    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"]
Oct 31 01:33:19.387494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387481    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0f4d9b96-cc50-4358-8571-174d1d6cef11" containerName="revoke-keys"
Oct 31 01:33:19.387553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387496    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4d9b96-cc50-4358-8571-174d1d6cef11" containerName="revoke-keys"
Oct 31 01:33:19.387553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387541    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy"
Oct 31 01:33:19.387553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387546    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy"
Oct 31 01:33:19.387675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387563    5419 cpu_manager.go:414] "containerMap: removing container" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy"
Oct 31 01:33:19.387675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387568    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy"
Oct 31 01:33:19.387675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387654    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44131ca-439b-431e-86b0-88edb2b5e9da" containerName="istio-proxy"
Oct 31 01:33:19.387675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387666    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="11ea7f89-4b8c-41d4-8107-f95ad9aa6f71" containerName="istio-proxy"
Oct 31 01:33:19.387675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.387671    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4d9b96-cc50-4358-8571-174d1d6cef11" containerName="revoke-keys"
Oct 31 01:33:19.392977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.392961    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.447276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.447251    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:33:19.449175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.449130    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"]
Oct 31 01:33:19.494190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.494161    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:33:19.494332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.494203    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:33:19.494427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.494414    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:33:19.517119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.517091    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bq9\" (UniqueName: \"kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.517364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.517344    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.517433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.517379    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.618774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.618741    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.619285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.618780    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.619285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.618841    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bq9\" (UniqueName: \"kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.619579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.619556    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.621301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.621281    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.675188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.675100    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bq9\" (UniqueName: \"kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9\") pod \"maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:19.707506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:19.707482    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:20.083940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:20.083913    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"]
Oct 31 01:33:20.084631 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:33:20.084602    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba04a873_ac39_4cfc_a0dc_bc2a49a348a1.slice/crio-33bbfd22fcb0722b791cc53f303876b1d9261f30689af5e49e416174f0fd2639 WatchSource:0}: Error finding container 33bbfd22fcb0722b791cc53f303876b1d9261f30689af5e49e416174f0fd2639: Status 404 returned error can't find the container with id 33bbfd22fcb0722b791cc53f303876b1d9261f30689af5e49e416174f0fd2639
Oct 31 01:33:20.414399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:20.414368    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl" event={"ID":"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1","Type":"ContainerStarted","Data":"f273156dc8e53a6b4c15715def32be7cd0ffd4b171c8e3456028d1d677ad3262"}
Oct 31 01:33:20.414602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:20.414420    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl" event={"ID":"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1","Type":"ContainerStarted","Data":"33bbfd22fcb0722b791cc53f303876b1d9261f30689af5e49e416174f0fd2639"}
Oct 31 01:33:20.419954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:20.419929    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"]
Oct 31 01:33:20.652196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:20.652075    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl" podStartSLOduration=1.6520620670000001 podStartE2EDuration="1.652062067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:20.53906394 +0000 UTC m=+1721.840020518" watchObservedRunningTime="2026-10-31 01:33:20.652062067 +0000 UTC m=+1721.953018652"
Oct 31 01:33:21.423369 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:21.423288    5419 generic.go:356] "Generic (PLEG): container finished" podID="ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" containerID="f273156dc8e53a6b4c15715def32be7cd0ffd4b171c8e3456028d1d677ad3262" exitCode=0
Oct 31 01:33:21.423506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:21.423370    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl" event={"ID":"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1","Type":"ContainerDied","Data":"f273156dc8e53a6b4c15715def32be7cd0ffd4b171c8e3456028d1d677ad3262"}
Oct 31 01:33:22.579013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.578992    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:22.724045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.724012    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca\") pod \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") "
Oct 31 01:33:22.724257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.724168    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca\") pod \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") "
Oct 31 01:33:22.724344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.724281    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9\" (UniqueName: \"kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9\") pod \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\" (UID: \"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1\") "
Oct 31 01:33:22.724610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.724579    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca" pod "ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" (UID: "ba04a873-ac39-4cfc-a0dc-bc2a49a348a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:22.724945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.724908    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:22.726579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.726548    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9" pod "ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" (UID: "ba04a873-ac39-4cfc-a0dc-bc2a49a348a1"). InnerVolumeSpecName "kube-api-access-l4bq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:22.726870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.726848    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca" pod "ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" (UID: "ba04a873-ac39-4cfc-a0dc-bc2a49a348a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:22.825760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.825721    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4bq9\" (UniqueName: \"kubernetes.io/projected/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-kube-api-access-l4bq9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:22.825760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:22.825752    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.437883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.437852    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"
Oct 31 01:33:23.438055 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.437858    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl" event={"ID":"ba04a873-ac39-4cfc-a0dc-bc2a49a348a1","Type":"ContainerDied","Data":"33bbfd22fcb0722b791cc53f303876b1d9261f30689af5e49e416174f0fd2639"}
Oct 31 01:33:23.438055 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.437980    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33bbfd22fcb0722b791cc53f303876b1d9261f30689af5e49e416174f0fd2639"
Oct 31 01:33:23.824428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.824405    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:33:23.941147 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941116    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941242    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941346    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941380    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941422    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941497    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941534    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941495    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:23.941617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941543    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.941785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941680    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941686    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.941785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941715    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data\") pod \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\" (UID: \"f0ab9da4-de4b-45a1-a08a-06f7def7d064\") "
Oct 31 01:33:23.941977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941896    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.942037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.941980    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.942575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.942546    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.942698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.942578    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.942698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.942595    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.942698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.942610    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.942698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.942625    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:23.943800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.943778    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "kube-api-access-g9l8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:23.943907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.943807    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:23.943907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.943804    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:23.943907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:23.943861    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo" pod "f0ab9da4-de4b-45a1-a08a-06f7def7d064" (UID: "f0ab9da4-de4b-45a1-a08a-06f7def7d064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:24.043567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.043487    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:24.043567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.043516    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9l8k\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-kube-api-access-g9l8k\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:24.043567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.043527    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:24.043567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.043536    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ab9da4-de4b-45a1-a08a-06f7def7d064-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:24.446977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.446886    5419 generic.go:356] "Generic (PLEG): container finished" podID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerID="c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0" exitCode=0
Oct 31 01:33:24.446977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.446940    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"
Oct 31 01:33:24.446977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.446970    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" event={"ID":"f0ab9da4-de4b-45a1-a08a-06f7def7d064","Type":"ContainerDied","Data":"c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0"}
Oct 31 01:33:24.447281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.447023    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929" event={"ID":"f0ab9da4-de4b-45a1-a08a-06f7def7d064","Type":"ContainerDied","Data":"cce418fe9fbedc2f14c29a4ed5b6b57ba5fbf78ce06750a726dc7e0a2e258315"}
Oct 31 01:33:24.447281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.447045    5419 scope.go:122] "RemoveContainer" containerID="c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0"
Oct 31 01:33:24.458496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.458479    5419 scope.go:122] "RemoveContainer" containerID="6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"
Oct 31 01:33:24.470868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.470848    5419 scope.go:122] "RemoveContainer" containerID="c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0"
Oct 31 01:33:24.471193 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:24.471172    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0\": container with ID starting with c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0 not found: ID does not exist" containerID="c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0"
Oct 31 01:33:24.471268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.471210    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0"} err="failed to get container status \"c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0\": rpc error: code = NotFound desc = could not find container \"c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0\": container with ID starting with c00e445b14d7203a52c974fa0b111934aee5dd984d82b7be2962a231561033a0 not found: ID does not exist"
Oct 31 01:33:24.471268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.471235    5419 scope.go:122] "RemoveContainer" containerID="6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"
Oct 31 01:33:24.471529 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:24.471510    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b\": container with ID starting with 6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b not found: ID does not exist" containerID="6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"
Oct 31 01:33:24.471569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.471535    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b"} err="failed to get container status \"6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b\": rpc error: code = NotFound desc = could not find container \"6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b\": container with ID starting with 6c495f1d6769ed425ddfe82d1f249de6e7ba297099a4a6623ba3755c1cbf351b not found: ID does not exist"
Oct 31 01:33:24.660695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.660656    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"]
Oct 31 01:33:24.709919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.709832    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-01cbd8-openshift-default-744b45f5f7-fw929"]
Oct 31 01:33:24.838065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:24.838021    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" path="/var/lib/kubelet/pods/f0ab9da4-de4b-45a1-a08a-06f7def7d064/volumes"
Oct 31 01:33:25.420760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.420707    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc" podUID="6f214e9f-cd7c-4497-bb9e-774a7d50827f" containerName="payload-processing" containerID="cri-o://e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c" gracePeriod=25
Oct 31 01:33:25.707490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.707464    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:33:25.763215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.763188    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume\") pod \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") "
Oct 31 01:33:25.763369 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.763308    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9\" (UniqueName: \"kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9\") pod \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\" (UID: \"6f214e9f-cd7c-4497-bb9e-774a7d50827f\") "
Oct 31 01:33:25.763559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.763538    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume" pod "6f214e9f-cd7c-4497-bb9e-774a7d50827f" (UID: "6f214e9f-cd7c-4497-bb9e-774a7d50827f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:25.763784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.763761    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f214e9f-cd7c-4497-bb9e-774a7d50827f-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:25.765507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.765486    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9" pod "6f214e9f-cd7c-4497-bb9e-774a7d50827f" (UID: "6f214e9f-cd7c-4497-bb9e-774a7d50827f"). InnerVolumeSpecName "kube-api-access-jcmr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:25.865312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:25.865278    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmr9\" (UniqueName: \"kubernetes.io/projected/6f214e9f-cd7c-4497-bb9e-774a7d50827f-kube-api-access-jcmr9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:26.463577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.463544    5419 generic.go:356] "Generic (PLEG): container finished" podID="6f214e9f-cd7c-4497-bb9e-774a7d50827f" containerID="e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c" exitCode=0
Oct 31 01:33:26.463785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.463589    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"
Oct 31 01:33:26.463785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.463633    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc" event={"ID":"6f214e9f-cd7c-4497-bb9e-774a7d50827f","Type":"ContainerDied","Data":"e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c"}
Oct 31 01:33:26.463785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.463685    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc" event={"ID":"6f214e9f-cd7c-4497-bb9e-774a7d50827f","Type":"ContainerDied","Data":"c3edc21805fe8a47673c9c95cf13bcc3f8fbdcc360978f3770f5ff6535bf04c9"}
Oct 31 01:33:26.463785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.463706    5419 scope.go:122] "RemoveContainer" containerID="e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c"
Oct 31 01:33:26.474512 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.474496    5419 scope.go:122] "RemoveContainer" containerID="e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c"
Oct 31 01:33:26.474772 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:26.474753    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c\": container with ID starting with e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c not found: ID does not exist" containerID="e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c"
Oct 31 01:33:26.474829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.474779    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c"} err="failed to get container status \"e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c\": rpc error: code = NotFound desc = could not find container \"e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c\": container with ID starting with e69c61bec811f302275dbb915d8c7731b4bbc661842479c8b9f17dcde76f220c not found: ID does not exist"
Oct 31 01:33:26.627707 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.627672    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"]
Oct 31 01:33:26.678025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.677989    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-42c96d-5696f4c5ff-bd6kc"]
Oct 31 01:33:26.851424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:26.851380    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f214e9f-cd7c-4497-bb9e-774a7d50827f" path="/var/lib/kubelet/pods/6f214e9f-cd7c-4497-bb9e-774a7d50827f/volumes"
Oct 31 01:33:28.980727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:28.980696    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"]
Oct 31 01:33:28.981216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:28.980890    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy" containerID="cri-o://0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5" gracePeriod=30
Oct 31 01:33:28.983804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:28.983774    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"]
Oct 31 01:33:28.983998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:28.983977    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy" containerID="cri-o://5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459" gracePeriod=30
Oct 31 01:33:29.031772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:29.031735    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"]
Oct 31 01:33:29.031980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:29.031943    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy" containerID="cri-o://75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a" gracePeriod=30
Oct 31 01:33:29.051054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:29.051020    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"]
Oct 31 01:33:29.051306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:29.051283    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" containerID="cri-o://7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251" gracePeriod=30
Oct 31 01:33:30.977869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:30.977654    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"]
Oct 31 01:33:33.432891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:33.432853    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:33:33.433297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:33.432928    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:33:34.369609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.369578    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:33:34.389775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.389747    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:33:34.393478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.393449    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:33:34.397797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.397730    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:33:34.452369 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.452333    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.452884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.452387    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.452884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.452424    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.452884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.452475    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.452884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.452663    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.453118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.452998    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.453118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453065    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.453334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453318    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.453399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453369    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.453460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453409    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.453460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453450    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.453570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453502    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.453624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453570    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.453755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453717    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.453845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453756    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:34.453845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453789    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.453845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453836    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.454006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453869    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.454006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453911    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.454006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453938    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.454006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453965    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.454006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.453991    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454010    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454022    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454095    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454127    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454190    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket\") pod \"b45bb8c6-dd67-4747-92d6-65f9522ed786\" (UID: \"b45bb8c6-dd67-4747-92d6-65f9522ed786\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454238    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454250    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454265    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454293    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454322    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454382    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454418    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454423    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454449    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data\") pod \"f5daad64-325f-486e-8a64-166579b9dac8\" (UID: \"f5daad64-325f-486e-8a64-166579b9dac8\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454499    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454549    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454590    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454622    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token\") pod \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\" (UID: \"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9\") "
Oct 31 01:33:34.454757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454728    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token\") pod \"b19466f9-e55d-490b-a30e-734467091321\" (UID: \"b19466f9-e55d-490b-a30e-734467091321\") "
Oct 31 01:33:34.455376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.455345    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.455376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.455366    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.455376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.455376    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.455549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.455385    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5daad64-325f-486e-8a64-166579b9dac8-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454588    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.454871    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.455197    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.455979    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.456196    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.456217    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.456580    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.456808    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.456838    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.457064    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.457563    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.457760    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.457940    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.458100    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.458308    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.458892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.458693    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "kube-api-access-q4hjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.459812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.459628    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:34.459812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.459690    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:34.460567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.460538    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.461280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.461248    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.461934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.461913    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:34.462175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462128    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.462688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462583    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:33:34.462688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462590    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy" pod "f5daad64-325f-486e-8a64-166579b9dac8" (UID: "f5daad64-325f-486e-8a64-166579b9dac8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.462688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462616    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "kube-api-access-cwkkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.462688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462646    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.462688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462680    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv" pod "b45bb8c6-dd67-4747-92d6-65f9522ed786" (UID: "b45bb8c6-dd67-4747-92d6-65f9522ed786"). InnerVolumeSpecName "kube-api-access-d47cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.462688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462689    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.463114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.462976    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "kube-api-access-c9gkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:34.463609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.463574    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy" pod "b19466f9-e55d-490b-a30e-734467091321" (UID: "b19466f9-e55d-490b-a30e-734467091321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.464008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.463987    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy" pod "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" (UID: "6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:34.551000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.550910    5419 generic.go:356] "Generic (PLEG): container finished" podID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerID="0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5" exitCode=0
Oct 31 01:33:34.551000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.550952    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"
Oct 31 01:33:34.551294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.550988    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" event={"ID":"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9","Type":"ContainerDied","Data":"0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5"}
Oct 31 01:33:34.551294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.551042    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2" event={"ID":"6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9","Type":"ContainerDied","Data":"c745895d334a8333a50599a913cc3482a6fb6f625834eb7e8581c38125e15ed9"}
Oct 31 01:33:34.551294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.551066    5419 scope.go:122] "RemoveContainer" containerID="0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5"
Oct 31 01:33:34.552683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.552657    5419 generic.go:356] "Generic (PLEG): container finished" podID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerID="5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459" exitCode=0
Oct 31 01:33:34.552840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.552710    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"
Oct 31 01:33:34.552840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.552756    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" event={"ID":"b45bb8c6-dd67-4747-92d6-65f9522ed786","Type":"ContainerDied","Data":"5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459"}
Oct 31 01:33:34.552840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.552811    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn" event={"ID":"b45bb8c6-dd67-4747-92d6-65f9522ed786","Type":"ContainerDied","Data":"61dfd67436121cd81582befbd4f7a0746cde97827dfb2af778878c37423293b6"}
Oct 31 01:33:34.559692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559667    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5daad64-325f-486e-8a64-166579b9dac8-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559695    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559711    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559725    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b45bb8c6-dd67-4747-92d6-65f9522ed786-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559739    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b19466f9-e55d-490b-a30e-734467091321-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559753    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559768    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559781    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559794    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b19466f9-e55d-490b-a30e-734467091321-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559809    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559822    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559835    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559848    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559861    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559875    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hjd\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-kube-api-access-q4hjd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559891    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9gkf\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-kube-api-access-c9gkf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559904    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559917    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b45bb8c6-dd67-4747-92d6-65f9522ed786-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559929    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5daad64-325f-486e-8a64-166579b9dac8-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559943    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559957    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwkkp\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-kube-api-access-cwkkp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559969    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559981    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b19466f9-e55d-490b-a30e-734467091321-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.559995    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560007    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5daad64-325f-486e-8a64-166579b9dac8-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560020    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560033    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560045    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560057    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.560056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560070    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19466f9-e55d-490b-a30e-734467091321-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.562107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560084    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b45bb8c6-dd67-4747-92d6-65f9522ed786-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.562107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.560098    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47cv\" (UniqueName: \"kubernetes.io/projected/b45bb8c6-dd67-4747-92d6-65f9522ed786-kube-api-access-d47cv\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:34.566777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.566577    5419 generic.go:356] "Generic (PLEG): container finished" podID="f5daad64-325f-486e-8a64-166579b9dac8" containerID="7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251" exitCode=0
Oct 31 01:33:34.566777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.566684    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"
Oct 31 01:33:34.566777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.566714    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" event={"ID":"f5daad64-325f-486e-8a64-166579b9dac8","Type":"ContainerDied","Data":"7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251"}
Oct 31 01:33:34.566777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.566770    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z" event={"ID":"f5daad64-325f-486e-8a64-166579b9dac8","Type":"ContainerDied","Data":"fcea05a450a1c4d761096f1923cbbad69703947f9cb2f78a2608d646639ce5ad"}
Oct 31 01:33:34.567639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.567621    5419 scope.go:122] "RemoveContainer" containerID="0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5"
Oct 31 01:33:34.568739 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:34.568605    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5\": container with ID starting with 0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5 not found: ID does not exist" containerID="0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5"
Oct 31 01:33:34.568739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.568639    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5"} err="failed to get container status \"0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5\": rpc error: code = NotFound desc = could not find container \"0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5\": container with ID starting with 0225422b74492079e137d2f740c05fcf479c600beded242ead6751f7bda925b5 not found: ID does not exist"
Oct 31 01:33:34.568739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.568662    5419 scope.go:122] "RemoveContainer" containerID="5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459"
Oct 31 01:33:34.570004 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.569979    5419 generic.go:356] "Generic (PLEG): container finished" podID="b19466f9-e55d-490b-a30e-734467091321" containerID="75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a" exitCode=0
Oct 31 01:33:34.571035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.570201    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" event={"ID":"b19466f9-e55d-490b-a30e-734467091321","Type":"ContainerDied","Data":"75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a"}
Oct 31 01:33:34.571035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.570252    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m" event={"ID":"b19466f9-e55d-490b-a30e-734467091321","Type":"ContainerDied","Data":"ccee567cebab4134d0e945a698d329ea2d841bd0215740c8827c27708c801055"}
Oct 31 01:33:34.571035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.570502    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"
Oct 31 01:33:34.582093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.582072    5419 scope.go:122] "RemoveContainer" containerID="5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459"
Oct 31 01:33:34.582564 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:34.582540    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459\": container with ID starting with 5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459 not found: ID does not exist" containerID="5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459"
Oct 31 01:33:34.582671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.582586    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459"} err="failed to get container status \"5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459\": rpc error: code = NotFound desc = could not find container \"5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459\": container with ID starting with 5d653c3f96f8d4e79efe09b6037b46542973b020830c10e0c4f10d6ca00f0459 not found: ID does not exist"
Oct 31 01:33:34.582671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.582611    5419 scope.go:122] "RemoveContainer" containerID="7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251"
Oct 31 01:33:34.594830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.594810    5419 scope.go:122] "RemoveContainer" containerID="7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251"
Oct 31 01:33:34.595081 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:34.595062    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251\": container with ID starting with 7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251 not found: ID does not exist" containerID="7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251"
Oct 31 01:33:34.595179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.595086    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251"} err="failed to get container status \"7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251\": rpc error: code = NotFound desc = could not find container \"7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251\": container with ID starting with 7107b6b5c891936c450bb0cf4c57500fe82068bc97b4309c9957322302cbb251 not found: ID does not exist"
Oct 31 01:33:34.595179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.595103    5419 scope.go:122] "RemoveContainer" containerID="75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a"
Oct 31 01:33:34.604657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.604643    5419 scope.go:122] "RemoveContainer" containerID="75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a"
Oct 31 01:33:34.604924 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:34.604908    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a\": container with ID starting with 75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a not found: ID does not exist" containerID="75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a"
Oct 31 01:33:34.604961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:34.604934    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a"} err="failed to get container status \"75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a\": rpc error: code = NotFound desc = could not find container \"75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a\": container with ID starting with 75f9ffd30fa9feaf526bc1af6fb0df54256cfad66284404fda00053bbeebfb6a not found: ID does not exist"
Oct 31 01:33:35.018665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.018621    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"]
Oct 31 01:33:35.021619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.021598    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-pd66m"]
Oct 31 01:33:35.119888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.119849    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"]
Oct 31 01:33:35.124220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.124196    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-5ncdn"]
Oct 31 01:33:35.231764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.231733    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"]
Oct 31 01:33:35.235826 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.235801    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-f6s8z"]
Oct 31 01:33:35.344181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.344064    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"]
Oct 31 01:33:35.347439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.347413    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-27jm2"]
Oct 31 01:33:35.978109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:35.978067    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5" podUID="2d635118-4d0f-48ef-a9d3-e0f39752c250" containerName="payload-pre-processing" containerID="cri-o://b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793" gracePeriod=25
Oct 31 01:33:36.248422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.248397    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:33:36.281654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.281624    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume\") pod \"2d635118-4d0f-48ef-a9d3-e0f39752c250\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") "
Oct 31 01:33:36.281793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.281774    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z\" (UniqueName: \"kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z\") pod \"2d635118-4d0f-48ef-a9d3-e0f39752c250\" (UID: \"2d635118-4d0f-48ef-a9d3-e0f39752c250\") "
Oct 31 01:33:36.282059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.282039    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume" pod "2d635118-4d0f-48ef-a9d3-e0f39752c250" (UID: "2d635118-4d0f-48ef-a9d3-e0f39752c250"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:36.282343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.282322    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d635118-4d0f-48ef-a9d3-e0f39752c250-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:36.284180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.284125    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z" pod "2d635118-4d0f-48ef-a9d3-e0f39752c250" (UID: "2d635118-4d0f-48ef-a9d3-e0f39752c250"). InnerVolumeSpecName "kube-api-access-r8b2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:36.383864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.383823    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8b2z\" (UniqueName: \"kubernetes.io/projected/2d635118-4d0f-48ef-a9d3-e0f39752c250-kube-api-access-r8b2z\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:36.589686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.589599    5419 generic.go:356] "Generic (PLEG): container finished" podID="2d635118-4d0f-48ef-a9d3-e0f39752c250" containerID="b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793" exitCode=0
Oct 31 01:33:36.589686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.589654    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"
Oct 31 01:33:36.589880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.589688    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5" event={"ID":"2d635118-4d0f-48ef-a9d3-e0f39752c250","Type":"ContainerDied","Data":"b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793"}
Oct 31 01:33:36.589880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.589742    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5" event={"ID":"2d635118-4d0f-48ef-a9d3-e0f39752c250","Type":"ContainerDied","Data":"7b506de4ad8500e44bbb9191fae4ddc3f5aac538b977c45bc95953621bffa8e6"}
Oct 31 01:33:36.589880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.589764    5419 scope.go:122] "RemoveContainer" containerID="b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793"
Oct 31 01:33:36.600233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.600215    5419 scope.go:122] "RemoveContainer" containerID="b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793"
Oct 31 01:33:36.600513 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:36.600494    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793\": container with ID starting with b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793 not found: ID does not exist" containerID="b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793"
Oct 31 01:33:36.600563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.600521    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793"} err="failed to get container status \"b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793\": rpc error: code = NotFound desc = could not find container \"b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793\": container with ID starting with b256042ebb5b802ab1a1257e090345287e7ef73e51da37e383f66a85cc11c793 not found: ID does not exist"
Oct 31 01:33:36.752099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.752066    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"]
Oct 31 01:33:36.801401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.801374    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14ca09-589f7f47b7-c9tf5"]
Oct 31 01:33:36.860211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.858523    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d635118-4d0f-48ef-a9d3-e0f39752c250" path="/var/lib/kubelet/pods/2d635118-4d0f-48ef-a9d3-e0f39752c250/volumes"
Oct 31 01:33:36.860211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.859239    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" path="/var/lib/kubelet/pods/6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9/volumes"
Oct 31 01:33:36.860211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.859872    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b19466f9-e55d-490b-a30e-734467091321" path="/var/lib/kubelet/pods/b19466f9-e55d-490b-a30e-734467091321/volumes"
Oct 31 01:33:36.860979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.860956    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" path="/var/lib/kubelet/pods/b45bb8c6-dd67-4747-92d6-65f9522ed786/volumes"
Oct 31 01:33:36.861953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:36.861926    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5daad64-325f-486e-8a64-166579b9dac8" path="/var/lib/kubelet/pods/f5daad64-325f-486e-8a64-166579b9dac8/volumes"
Oct 31 01:33:38.336856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.336822    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"]
Oct 31 01:33:38.337417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337401    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" containerName="revoke-keys"
Oct 31 01:33:38.337458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337418    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" containerName="revoke-keys"
Oct 31 01:33:38.337458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337439    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy"
Oct 31 01:33:38.337458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337445    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy"
Oct 31 01:33:38.337458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337454    5419 cpu_manager.go:414] "containerMap: removing container" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337459    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337473    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337478    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337485    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337490    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337501    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2d635118-4d0f-48ef-a9d3-e0f39752c250" containerName="payload-pre-processing"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337506    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d635118-4d0f-48ef-a9d3-e0f39752c250" containerName="payload-pre-processing"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337514    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337518    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337524    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337528    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337547    5419 cpu_manager.go:414] "containerMap: removing container" podUID="6f214e9f-cd7c-4497-bb9e-774a7d50827f" containerName="payload-processing"
Oct 31 01:33:38.337596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337551    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f214e9f-cd7c-4497-bb9e-774a7d50827f" containerName="payload-processing"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337628    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d635118-4d0f-48ef-a9d3-e0f39752c250" containerName="payload-pre-processing"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337636    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5daad64-325f-486e-8a64-166579b9dac8" containerName="istio-proxy"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337642    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337650    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f214e9f-cd7c-4497-bb9e-774a7d50827f" containerName="payload-processing"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337658    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" containerName="revoke-keys"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337664    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45bb8c6-dd67-4747-92d6-65f9522ed786" containerName="istio-proxy"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337669    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4c5e8b-a0aa-4df4-ba5d-fa42e690eef9" containerName="istio-proxy"
Oct 31 01:33:38.337951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.337675    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b19466f9-e55d-490b-a30e-734467091321" containerName="istio-proxy"
Oct 31 01:33:38.342343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.342327    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.397041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.397012    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"]
Oct 31 01:33:38.421588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.421558    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbdxh\" (UniqueName: \"kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.421786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.421767    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.421834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.421821    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.442696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.442664    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:33:38.442696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.442678    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:33:38.442861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.442700    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:33:38.443787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.443764    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:33:38.523016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.522973    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.523260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.523046    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.523260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.523072    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbdxh\" (UniqueName: \"kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.523723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.523702    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.525797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.525777    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.579534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.579498    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbdxh\" (UniqueName: \"kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh\") pod \"maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:38.656381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:38.656307    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:39.031737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.031712    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"]
Oct 31 01:33:39.033842 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:33:39.033803    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e377ae_7ea9_4555_8c5b_49e5fc4476e0.slice/crio-336030bd237b9298f6417393f1bdd8309fb1143f9fa78570189c191a2cda4bd2 WatchSource:0}: Error finding container 336030bd237b9298f6417393f1bdd8309fb1143f9fa78570189c191a2cda4bd2: Status 404 returned error can't find the container with id 336030bd237b9298f6417393f1bdd8309fb1143f9fa78570189c191a2cda4bd2
Oct 31 01:33:39.619064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.618976    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd" event={"ID":"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0","Type":"ContainerStarted","Data":"bdd79342e74701c7008d61b4d3da1d80ada7d4e72f210f289d57d55ccc990523"}
Oct 31 01:33:39.619064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.619046    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd" event={"ID":"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0","Type":"ContainerStarted","Data":"336030bd237b9298f6417393f1bdd8309fb1143f9fa78570189c191a2cda4bd2"}
Oct 31 01:33:39.731754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.731701    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd" podStartSLOduration=1.73168761 podStartE2EDuration="1.73168761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:39.731216582 +0000 UTC m=+1741.032173183" watchObservedRunningTime="2026-10-31 01:33:39.73168761 +0000 UTC m=+1741.032644186"
Oct 31 01:33:39.856887 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:39.856860    5419 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e377ae_7ea9_4555_8c5b_49e5fc4476e0.slice/crio-bdd79342e74701c7008d61b4d3da1d80ada7d4e72f210f289d57d55ccc990523.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:33:39.906682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.906657    5419 scope.go:122] "RemoveContainer" containerID="b2ab9f9e067dcf766cf70f6210001d98cb67b9eec007f822d32fe5b1d04afb7b"
Oct 31 01:33:39.949733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.949714    5419 scope.go:122] "RemoveContainer" containerID="3ffdcd5c708b2695c417cbcf100d5b1b80a2dcae0e9aa26d7675a73ef6b4491e"
Oct 31 01:33:39.958598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.958577    5419 scope.go:122] "RemoveContainer" containerID="5fe6e921de9f8b8fef0eabef708b45bcac9f81aacd12cbc9276347141502f40b"
Oct 31 01:33:39.967408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:39.967390    5419 scope.go:122] "RemoveContainer" containerID="0c68b0c31d02a5551737c152ac69cc07605f862f909fd6980c07dbdac2825bf4"
Oct 31 01:33:40.419199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:40.419162    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:33:40.627776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:40.627741    5419 generic.go:356] "Generic (PLEG): container finished" podID="e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" containerID="bdd79342e74701c7008d61b4d3da1d80ada7d4e72f210f289d57d55ccc990523" exitCode=0
Oct 31 01:33:40.628188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:40.627789    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd" event={"ID":"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0","Type":"ContainerDied","Data":"bdd79342e74701c7008d61b4d3da1d80ada7d4e72f210f289d57d55ccc990523"}
Oct 31 01:33:41.789045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.789003    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:41.865936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.865904    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca\") pod \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") "
Oct 31 01:33:41.866095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.866007    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh\" (UniqueName: \"kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh\") pod \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") "
Oct 31 01:33:41.866095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.866037    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca\") pod \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\" (UID: \"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0\") "
Oct 31 01:33:41.866264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.866241    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca" pod "e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" (UID: "e8e377ae-7ea9-4555-8c5b-49e5fc4476e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:41.866557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.866537    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:41.868152 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.868122    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh" pod "e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" (UID: "e8e377ae-7ea9-4555-8c5b-49e5fc4476e0"). InnerVolumeSpecName "kube-api-access-sbdxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:41.868313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.868296    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca" pod "e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" (UID: "e8e377ae-7ea9-4555-8c5b-49e5fc4476e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:41.967350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.967275    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbdxh\" (UniqueName: \"kubernetes.io/projected/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-kube-api-access-sbdxh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:41.967350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:41.967301    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:42.641034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:42.641006    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"
Oct 31 01:33:42.641254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:42.641009    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd" event={"ID":"e8e377ae-7ea9-4555-8c5b-49e5fc4476e0","Type":"ContainerDied","Data":"336030bd237b9298f6417393f1bdd8309fb1143f9fa78570189c191a2cda4bd2"}
Oct 31 01:33:42.641254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:42.641148    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336030bd237b9298f6417393f1bdd8309fb1143f9fa78570189c191a2cda4bd2"
Oct 31 01:33:44.584230 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:44.584187    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:33:45.378443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.378403    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"]
Oct 31 01:33:45.378982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.378966    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" containerName="revoke-keys"
Oct 31 01:33:45.378982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.378980    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" containerName="revoke-keys"
Oct 31 01:33:45.379166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.379069    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" containerName="revoke-keys"
Oct 31 01:33:45.379166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.379080    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ab9da4-de4b-45a1-a08a-06f7def7d064" containerName="istio-proxy"
Oct 31 01:33:45.382613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.382594    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.447326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.441620    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-5e916d-openshift-default-dockercfg-jlpnq\""
Oct 31 01:33:45.532240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.532188    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.553917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.553876    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.554292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.554263    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.554589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.554561    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.554830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.554812    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.554999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.554982    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4d4\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.555213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.555197    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.555377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.555362    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.557365 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.557341    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.589353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.589316    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"]
Oct 31 01:33:45.669155 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.669044    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.669155 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.669106    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4d4\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.669495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.669193    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.672528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.669604    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.686592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.673908    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.686870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.674155    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.687012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.669225    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.687404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.687334    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.688065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.688045    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.688764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.688743    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.689718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.689699    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.693108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.690451    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.693669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.689610    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.693669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.687926    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.696293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.695302    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.698874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.698837    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.730675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.730621    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4d4\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.745584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.745552    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token\") pod \"e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") " pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:45.996431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:45.996381    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:46.437352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.437320    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"]
Oct 31 01:33:46.438516 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:33:46.438486    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a75745b_0605_44fa_bc58_7fcc8fd51dfe.slice/crio-45762e627fa74796cc416fb95703d0fc7af1f44efcbb1df69dac6461c9228a2a WatchSource:0}: Error finding container 45762e627fa74796cc416fb95703d0fc7af1f44efcbb1df69dac6461c9228a2a: Status 404 returned error can't find the container with id 45762e627fa74796cc416fb95703d0fc7af1f44efcbb1df69dac6461c9228a2a
Oct 31 01:33:46.441314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.441282    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:33:46.441403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.441357    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:33:46.441403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.441386    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:33:46.709077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.708994    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" event={"ID":"4a75745b-0605-44fa-bc58-7fcc8fd51dfe","Type":"ContainerStarted","Data":"742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"}
Oct 31 01:33:46.709077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.709045    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" event={"ID":"4a75745b-0605-44fa-bc58-7fcc8fd51dfe","Type":"ContainerStarted","Data":"45762e627fa74796cc416fb95703d0fc7af1f44efcbb1df69dac6461c9228a2a"}
Oct 31 01:33:46.828869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.828817    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podStartSLOduration=1.828803483 podStartE2EDuration="1.828803483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:46.827795231 +0000 UTC m=+1748.128751831" watchObservedRunningTime="2026-10-31 01:33:46.828803483 +0000 UTC m=+1748.129760060"
Oct 31 01:33:46.998413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:46.998375    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:48.002045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:48.002014    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:48.721762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:48.721731    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:48.722811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:48.722790    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:33:49.822148 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:49.822094    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"]
Oct 31 01:33:54.133742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.133710    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"]
Oct 31 01:33:54.138020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.137992    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.195270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.195241    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"]
Oct 31 01:33:54.301337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.299012    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-5e916d-dockercfg-hpsld\""
Oct 31 01:33:54.301337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.299085    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-5e916d\""
Oct 31 01:33:54.349853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.349601    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6nz\" (UniqueName: \"kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz\") pod \"payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.350442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.350413    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.423039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.415273    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"]
Oct 31 01:33:54.432251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.431234    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"]
Oct 31 01:33:54.432251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.431361    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.464328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.464292    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.464494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.464435    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.464494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.464474    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srv2n\" (UniqueName: \"kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n\") pod \"payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.464626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.464505    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6nz\" (UniqueName: \"kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz\") pod \"payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.465023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.464996    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.521272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.521246    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6nz\" (UniqueName: \"kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz\") pod \"payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.565906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.565871    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.566054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.565918    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srv2n\" (UniqueName: \"kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n\") pod \"payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.566611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.566572    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.622243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.622205    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srv2n\" (UniqueName: \"kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n\") pod \"payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.745669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.745644    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:54.756103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.756074    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:54.826626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:54.822579    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9" podUID="7f72ac5a-9e67-4e4b-8dff-743d048811f9" containerName="payload-processing" containerID="cri-o://f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920" gracePeriod=25
Oct 31 01:33:55.077311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.077285    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:33:55.126355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.126331    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"]
Oct 31 01:33:55.127524 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:33:55.127502    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7c7220_c681_4c53_b18e_05ff582dcf0d.slice/crio-22fd5dc050b4985035dcb7dabbfd4ab226de1d73ddff40c181f089f56b00138d WatchSource:0}: Error finding container 22fd5dc050b4985035dcb7dabbfd4ab226de1d73ddff40c181f089f56b00138d: Status 404 returned error can't find the container with id 22fd5dc050b4985035dcb7dabbfd4ab226de1d73ddff40c181f089f56b00138d
Oct 31 01:33:55.173355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.173326    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6\" (UniqueName: \"kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6\") pod \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") "
Oct 31 01:33:55.173785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.173581    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume\") pod \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\" (UID: \"7f72ac5a-9e67-4e4b-8dff-743d048811f9\") "
Oct 31 01:33:55.173948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.173926    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume" pod "7f72ac5a-9e67-4e4b-8dff-743d048811f9" (UID: "7f72ac5a-9e67-4e4b-8dff-743d048811f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:33:55.174104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.174090    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f72ac5a-9e67-4e4b-8dff-743d048811f9-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:55.175757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.175517    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6" pod "7f72ac5a-9e67-4e4b-8dff-743d048811f9" (UID: "7f72ac5a-9e67-4e4b-8dff-743d048811f9"). InnerVolumeSpecName "kube-api-access-gllw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:55.177273 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:33:55.177247    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32ee054_d11f_4913_bc14_3cd904576f22.slice/crio-a0afd4089d481d07f5902f9fa5de3bcfff84646357246b7becc6c40f866e40e9 WatchSource:0}: Error finding container a0afd4089d481d07f5902f9fa5de3bcfff84646357246b7becc6c40f866e40e9: Status 404 returned error can't find the container with id a0afd4089d481d07f5902f9fa5de3bcfff84646357246b7becc6c40f866e40e9
Oct 31 01:33:55.177496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.177474    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"]
Oct 31 01:33:55.274828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.274801    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gllw6\" (UniqueName: \"kubernetes.io/projected/7f72ac5a-9e67-4e4b-8dff-743d048811f9-kube-api-access-gllw6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:33:55.783688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.783593    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw" event={"ID":"f32ee054-d11f-4913-bc14-3cd904576f22","Type":"ContainerStarted","Data":"34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969"}
Oct 31 01:33:55.783688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.783658    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw" event={"ID":"f32ee054-d11f-4913-bc14-3cd904576f22","Type":"ContainerStarted","Data":"a0afd4089d481d07f5902f9fa5de3bcfff84646357246b7becc6c40f866e40e9"}
Oct 31 01:33:55.783688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.783686    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:33:55.785463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.785427    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx" event={"ID":"ae7c7220-c681-4c53-b18e-05ff582dcf0d","Type":"ContainerStarted","Data":"b5c699ea8902e5ad26d943dac0f4c920bf83d45d9086c38b80f6eba186665855"}
Oct 31 01:33:55.785624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.785485    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx" event={"ID":"ae7c7220-c681-4c53-b18e-05ff582dcf0d","Type":"ContainerStarted","Data":"22fd5dc050b4985035dcb7dabbfd4ab226de1d73ddff40c181f089f56b00138d"}
Oct 31 01:33:55.785624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.785515    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:33:55.786768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.786742    5419 generic.go:356] "Generic (PLEG): container finished" podID="7f72ac5a-9e67-4e4b-8dff-743d048811f9" containerID="f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920" exitCode=0
Oct 31 01:33:55.786858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.786780    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"
Oct 31 01:33:55.786858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.786816    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9" event={"ID":"7f72ac5a-9e67-4e4b-8dff-743d048811f9","Type":"ContainerDied","Data":"f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920"}
Oct 31 01:33:55.786940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.786857    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9" event={"ID":"7f72ac5a-9e67-4e4b-8dff-743d048811f9","Type":"ContainerDied","Data":"fa1f07a0187cf71996a49288f798f8308d5b0bdc1ffbacf7455e9112afa37652"}
Oct 31 01:33:55.786940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.786880    5419 scope.go:122] "RemoveContainer" containerID="f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920"
Oct 31 01:33:55.797095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.797077    5419 scope.go:122] "RemoveContainer" containerID="f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920"
Oct 31 01:33:55.797403 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:33:55.797379    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920\": container with ID starting with f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920 not found: ID does not exist" containerID="f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920"
Oct 31 01:33:55.797488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.797412    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920"} err="failed to get container status \"f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920\": rpc error: code = NotFound desc = could not find container \"f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920\": container with ID starting with f7ab719ca3ef96453c580c6ef5e99497501ce3d0fa1c050ac180b2eefc26c920 not found: ID does not exist"
Oct 31 01:33:55.898790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:55.898744    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw" podStartSLOduration=1.898731593 podStartE2EDuration="1.898731593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:55.897564758 +0000 UTC m=+1757.198521314" watchObservedRunningTime="2026-10-31 01:33:55.898731593 +0000 UTC m=+1757.199688170"
Oct 31 01:33:56.026737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:56.026692    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"]
Oct 31 01:33:56.033116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:56.032969    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c70e7-5cf6795dc5-4jsj9"]
Oct 31 01:33:56.125255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:56.125160    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx" podStartSLOduration=2.125129425 podStartE2EDuration="2.125129425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:56.124261515 +0000 UTC m=+1757.425218116" watchObservedRunningTime="2026-10-31 01:33:56.125129425 +0000 UTC m=+1757.426086001"
Oct 31 01:33:56.837506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:56.837477    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f72ac5a-9e67-4e4b-8dff-743d048811f9" path="/var/lib/kubelet/pods/7f72ac5a-9e67-4e4b-8dff-743d048811f9/volumes"
Oct 31 01:33:59.328665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:59.328621    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"]
Oct 31 01:33:59.331227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:33:59.331205    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7eea9-mxc7p"]
Oct 31 01:34:00.839396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:00.839365    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2a73ad-d213-4aea-9dad-9a5923b4fd9e" path="/var/lib/kubelet/pods/7d2a73ad-d213-4aea-9dad-9a5923b4fd9e/volumes"
Oct 31 01:34:06.797562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:06.797534    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 01:34:06.798018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:06.797598    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 01:34:14.583433 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:34:14.583396    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:34:25.388672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:25.388639    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"]
Oct 31 01:34:25.437566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:25.437539    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-42c96d-5q9cr"]
Oct 31 01:34:26.838003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:26.837970    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4d9b96-cc50-4358-8571-174d1d6cef11" path="/var/lib/kubelet/pods/0f4d9b96-cc50-4358-8571-174d1d6cef11/volumes"
Oct 31 01:34:27.044079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:27.044045    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"]
Oct 31 01:34:27.056787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:27.056759    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3a06cb-gnqk8"]
Oct 31 01:34:27.800619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:27.800582    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:34:28.837405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:28.837366    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf46c21-01a3-4114-b531-cf64bfe7d7c2" path="/var/lib/kubelet/pods/8bf46c21-01a3-4114-b531-cf64bfe7d7c2/volumes"
Oct 31 01:34:34.111883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.111839    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"]
Oct 31 01:34:34.112794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.112771    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7f72ac5a-9e67-4e4b-8dff-743d048811f9" containerName="payload-processing"
Oct 31 01:34:34.112858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.112797    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f72ac5a-9e67-4e4b-8dff-743d048811f9" containerName="payload-processing"
Oct 31 01:34:34.113009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.112993    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f72ac5a-9e67-4e4b-8dff-743d048811f9" containerName="payload-processing"
Oct 31 01:34:34.117395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.117373    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.168561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.168533    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"]
Oct 31 01:34:34.251935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.251886    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.252305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.252280    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkvz\" (UniqueName: \"kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.252522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.252500    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.299174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.266362    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:34:34.299174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.266421    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:34:34.299174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.266510    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:34:34.299174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.267171    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:34:34.354743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.354693    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkvz\" (UniqueName: \"kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.355116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.355099    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.355330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.355313    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.355957 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.355932    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.359323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.359301    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.411332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.411255    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkvz\" (UniqueName: \"kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz\") pod \"maas-api-revoke-keys-e2e-api-b-bc2046-jm68d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.432821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.432791    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:34.806443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:34.806420    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"]
Oct 31 01:34:34.807942 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:34:34.807914    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b133c1_b827_4d38_9845_01d98e723e9d.slice/crio-2ab56bc0847260f2bbf64039f5f51d5424a60a9be8301a906e409ca86fa6189d WatchSource:0}: Error finding container 2ab56bc0847260f2bbf64039f5f51d5424a60a9be8301a906e409ca86fa6189d: Status 404 returned error can't find the container with id 2ab56bc0847260f2bbf64039f5f51d5424a60a9be8301a906e409ca86fa6189d
Oct 31 01:34:35.342949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:35.342848    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d" event={"ID":"41b133c1-b827-4d38-9845-01d98e723e9d","Type":"ContainerStarted","Data":"4d8b16fd432a2820354d60c35dd3aba1a723c91b80ac4f2d6dfaee0e9d37cf1b"}
Oct 31 01:34:35.342949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:35.342904    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d" event={"ID":"41b133c1-b827-4d38-9845-01d98e723e9d","Type":"ContainerStarted","Data":"2ab56bc0847260f2bbf64039f5f51d5424a60a9be8301a906e409ca86fa6189d"}
Oct 31 01:34:35.460193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:35.460123    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d" podStartSLOduration=1.460108458 podStartE2EDuration="1.460108458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:35.457274649 +0000 UTC m=+1796.758231221" watchObservedRunningTime="2026-10-31 01:34:35.460108458 +0000 UTC m=+1796.761065035"
Oct 31 01:34:36.351685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:36.351651    5419 generic.go:356] "Generic (PLEG): container finished" podID="41b133c1-b827-4d38-9845-01d98e723e9d" containerID="4d8b16fd432a2820354d60c35dd3aba1a723c91b80ac4f2d6dfaee0e9d37cf1b" exitCode=0
Oct 31 01:34:36.352086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:36.351747    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d" event={"ID":"41b133c1-b827-4d38-9845-01d98e723e9d","Type":"ContainerDied","Data":"4d8b16fd432a2820354d60c35dd3aba1a723c91b80ac4f2d6dfaee0e9d37cf1b"}
Oct 31 01:34:37.500905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.500883    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:37.548185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.538980    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca\") pod \"41b133c1-b827-4d38-9845-01d98e723e9d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") "
Oct 31 01:34:37.548185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.539067    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca\") pod \"41b133c1-b827-4d38-9845-01d98e723e9d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") "
Oct 31 01:34:37.548185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.539164    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz\" (UniqueName: \"kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz\") pod \"41b133c1-b827-4d38-9845-01d98e723e9d\" (UID: \"41b133c1-b827-4d38-9845-01d98e723e9d\") "
Oct 31 01:34:37.557502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.553769    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca" pod "41b133c1-b827-4d38-9845-01d98e723e9d" (UID: "41b133c1-b827-4d38-9845-01d98e723e9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:37.570671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.570617    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca" pod "41b133c1-b827-4d38-9845-01d98e723e9d" (UID: "41b133c1-b827-4d38-9845-01d98e723e9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:34:37.570840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.570677    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz" pod "41b133c1-b827-4d38-9845-01d98e723e9d" (UID: "41b133c1-b827-4d38-9845-01d98e723e9d"). InnerVolumeSpecName "kube-api-access-cpkvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:37.640524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.640448    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.640524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.640477    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41b133c1-b827-4d38-9845-01d98e723e9d-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:37.640524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:37.640488    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpkvz\" (UniqueName: \"kubernetes.io/projected/41b133c1-b827-4d38-9845-01d98e723e9d-kube-api-access-cpkvz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:38.365639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:38.365608    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"
Oct 31 01:34:38.365825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:38.365645    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d" event={"ID":"41b133c1-b827-4d38-9845-01d98e723e9d","Type":"ContainerDied","Data":"2ab56bc0847260f2bbf64039f5f51d5424a60a9be8301a906e409ca86fa6189d"}
Oct 31 01:34:38.365825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:38.365701    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab56bc0847260f2bbf64039f5f51d5424a60a9be8301a906e409ca86fa6189d"
Oct 31 01:34:44.583234 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:34:44.583197    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:34:45.232036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:45.232001    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"]
Oct 31 01:34:50.232499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.232395    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg" podUID="0ab31518-f838-4f11-a6fa-0c64f28ae9cb" containerName="payload-pre-processing" containerID="cri-o://c66c787b0d219710afb1cb16a6ed20f6dbf8394ed13b1abdacbde4780014c2e6" gracePeriod=25
Oct 31 01:34:50.446959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.446928    5419 generic.go:356] "Generic (PLEG): container finished" podID="0ab31518-f838-4f11-a6fa-0c64f28ae9cb" containerID="c66c787b0d219710afb1cb16a6ed20f6dbf8394ed13b1abdacbde4780014c2e6" exitCode=0
Oct 31 01:34:50.447182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.447000    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg" event={"ID":"0ab31518-f838-4f11-a6fa-0c64f28ae9cb","Type":"ContainerDied","Data":"c66c787b0d219710afb1cb16a6ed20f6dbf8394ed13b1abdacbde4780014c2e6"}
Oct 31 01:34:50.491977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.491915    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:34:50.672286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.672254    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm\" (UniqueName: \"kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm\") pod \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") "
Oct 31 01:34:50.672453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.672440    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume\") pod \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\" (UID: \"0ab31518-f838-4f11-a6fa-0c64f28ae9cb\") "
Oct 31 01:34:50.672866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.672836    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume" pod "0ab31518-f838-4f11-a6fa-0c64f28ae9cb" (UID: "0ab31518-f838-4f11-a6fa-0c64f28ae9cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:50.673039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.673023    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:50.674602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.674579    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm" pod "0ab31518-f838-4f11-a6fa-0c64f28ae9cb" (UID: "0ab31518-f838-4f11-a6fa-0c64f28ae9cb"). InnerVolumeSpecName "kube-api-access-w8trm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:50.775218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:50.775100    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8trm\" (UniqueName: \"kubernetes.io/projected/0ab31518-f838-4f11-a6fa-0c64f28ae9cb-kube-api-access-w8trm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:34:51.455459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:51.455428    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"
Oct 31 01:34:51.455998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:51.455462    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg" event={"ID":"0ab31518-f838-4f11-a6fa-0c64f28ae9cb","Type":"ContainerDied","Data":"053fed798502f8d941372920424098a2799d4818cb1d67abb18b2c8982ffaada"}
Oct 31 01:34:51.455998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:51.455526    5419 scope.go:122] "RemoveContainer" containerID="c66c787b0d219710afb1cb16a6ed20f6dbf8394ed13b1abdacbde4780014c2e6"
Oct 31 01:34:51.617343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:51.617307    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"]
Oct 31 01:34:51.666439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:51.666403    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bc2046-686fddbb8c-mrwqg"]
Oct 31 01:34:52.837839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:52.837800    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab31518-f838-4f11-a6fa-0c64f28ae9cb" path="/var/lib/kubelet/pods/0ab31518-f838-4f11-a6fa-0c64f28ae9cb/volumes"
Oct 31 01:34:56.326199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:56.326161    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"]
Oct 31 01:34:56.326766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:56.326472    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy" containerID="cri-o://3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649" gracePeriod=30
Oct 31 01:34:57.949309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:57.949276    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"]
Oct 31 01:34:58.003129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:58.003094    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c70e7-wkqrd"]
Oct 31 01:34:58.837545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:34:58.837512    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e377ae-7ea9-4555-8c5b-49e5fc4476e0" path="/var/lib/kubelet/pods/e8e377ae-7ea9-4555-8c5b-49e5fc4476e0/volumes"
Oct 31 01:35:01.605274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.605217    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:35:01.623922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.623886    5419 generic.go:356] "Generic (PLEG): container finished" podID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerID="3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649" exitCode=0
Oct 31 01:35:01.624057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.623941    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"
Oct 31 01:35:01.624057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.623975    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" event={"ID":"4e787433-9dc5-4e38-912f-b14630d9dda4","Type":"ContainerDied","Data":"3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649"}
Oct 31 01:35:01.624057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.624034    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7" event={"ID":"4e787433-9dc5-4e38-912f-b14630d9dda4","Type":"ContainerDied","Data":"a80a42253ab3220fe4760a219c5c6023b90b075000dac5de8b4b68e6a8d91ad2"}
Oct 31 01:35:01.624231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.624058    5419 scope.go:122] "RemoveContainer" containerID="3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649"
Oct 31 01:35:01.633759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.633736    5419 scope.go:122] "RemoveContainer" containerID="134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"
Oct 31 01:35:01.645284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.644258    5419 scope.go:122] "RemoveContainer" containerID="3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649"
Oct 31 01:35:01.645284 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:01.644802    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649\": container with ID starting with 3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649 not found: ID does not exist" containerID="3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649"
Oct 31 01:35:01.645284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.644836    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649"} err="failed to get container status \"3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649\": rpc error: code = NotFound desc = could not find container \"3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649\": container with ID starting with 3945e38a13e0b6433430fdedf054c1872ea26904e6f6448a02e3baffbc1ba649 not found: ID does not exist"
Oct 31 01:35:01.645284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.644859    5419 scope.go:122] "RemoveContainer" containerID="134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"
Oct 31 01:35:01.645284 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:01.645243    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054\": container with ID starting with 134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054 not found: ID does not exist" containerID="134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"
Oct 31 01:35:01.645284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.645269    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054"} err="failed to get container status \"134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054\": rpc error: code = NotFound desc = could not find container \"134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054\": container with ID starting with 134b9a7656dd09be685ffb178eeea69ee188843ce351b1131bf28270bd389054 not found: ID does not exist"
Oct 31 01:35:01.655731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.655708    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.655851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.655762    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.655851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.655791    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.655851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.655837    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.656028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.655893    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.656028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.655920    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.656161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656110    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.656230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656195    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:01.656290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656233    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.656290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656256    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:01.656396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656332    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs\") pod \"4e787433-9dc5-4e38-912f-b14630d9dda4\" (UID: \"4e787433-9dc5-4e38-912f-b14630d9dda4\") "
Oct 31 01:35:01.656495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656471    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:01.656495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656484    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:01.656660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656637    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:01.656942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656919    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.657058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656949    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.657058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.656968    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787433-9dc5-4e38-912f-b14630d9dda4-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.657159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.657068    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.657214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.657185    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.658554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.658524    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:01.658685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.658672    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:35:01.658755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.658699    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:01.659004 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.658983    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb" pod "4e787433-9dc5-4e38-912f-b14630d9dda4" (UID: "4e787433-9dc5-4e38-912f-b14630d9dda4"). InnerVolumeSpecName "kube-api-access-6kbvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:01.758488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.758452    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbvb\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-kube-api-access-6kbvb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.758488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.758482    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.758488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.758495    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:01.758707 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:01.758508    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787433-9dc5-4e38-912f-b14630d9dda4-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:02.045720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:02.045645    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"]
Oct 31 01:35:02.050991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:02.050956    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-9qlh7"]
Oct 31 01:35:02.863744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:02.863708    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" path="/var/lib/kubelet/pods/4e787433-9dc5-4e38-912f-b14630d9dda4/volumes"
Oct 31 01:35:03.683176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.683121    5419 generic.go:356] "Generic (PLEG): container finished" podID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerID="742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae" exitCode=0
Oct 31 01:35:03.683400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.683184    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" event={"ID":"4a75745b-0605-44fa-bc58-7fcc8fd51dfe","Type":"ContainerDied","Data":"742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"}
Oct 31 01:35:03.683621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.683607    5419 scope.go:122] "RemoveContainer" containerID="742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"
Oct 31 01:35:03.685222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.685183    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:03.685302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.685271    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:03.685339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.685308    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:03.721970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:03.721940    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:35:04.451864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:04.451833    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"]
Oct 31 01:35:04.454733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:04.454707    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-14ca09-fvjnl"]
Oct 31 01:35:04.692982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:04.692948    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" event={"ID":"4a75745b-0605-44fa-bc58-7fcc8fd51dfe","Type":"ContainerStarted","Data":"6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22"}
Oct 31 01:35:04.766525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:04.766492    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"]
Oct 31 01:35:04.766806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:04.766762    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" containerID="cri-o://073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674" gracePeriod=30
Oct 31 01:35:04.843509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:04.843471    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba04a873-ac39-4cfc-a0dc-bc2a49a348a1" path="/var/lib/kubelet/pods/ba04a873-ac39-4cfc-a0dc-bc2a49a348a1/volumes"
Oct 31 01:35:05.002673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.002640    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:35:05.002940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.002885    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:35:05.003002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.002949    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:35:05.701763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.701728    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"]
Oct 31 01:35:05.702606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702583    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy"
Oct 31 01:35:05.702765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702609    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy"
Oct 31 01:35:05.702765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702680    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0ab31518-f838-4f11-a6fa-0c64f28ae9cb" containerName="payload-pre-processing"
Oct 31 01:35:05.702765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702689    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab31518-f838-4f11-a6fa-0c64f28ae9cb" containerName="payload-pre-processing"
Oct 31 01:35:05.702765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702704    5419 cpu_manager.go:414] "containerMap: removing container" podUID="41b133c1-b827-4d38-9845-01d98e723e9d" containerName="revoke-keys"
Oct 31 01:35:05.702765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702712    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b133c1-b827-4d38-9845-01d98e723e9d" containerName="revoke-keys"
Oct 31 01:35:05.702968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702845    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy"
Oct 31 01:35:05.702968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702863    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab31518-f838-4f11-a6fa-0c64f28ae9cb" containerName="payload-pre-processing"
Oct 31 01:35:05.702968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.702874    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b133c1-b827-4d38-9845-01d98e723e9d" containerName="revoke-keys"
Oct 31 01:35:05.707422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.707400    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.751977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.751943    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"]
Oct 31 01:35:05.761177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.759082    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:35:05.761177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.759191    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:35:05.763041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.763010    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vrd\" (UniqueName: \"kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.763263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.763243    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.763349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.763282    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.766921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.766891    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:35:05.766921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.766900    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:35:05.865356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.865319    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.865542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.865366    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.865669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.865535    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vrd\" (UniqueName: \"kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.866128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.866101    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.868057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.868034    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.922088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.922050    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vrd\" (UniqueName: \"kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd\") pod \"maas-api-revoke-keys-e2e-api-b-57ea84-79fpw\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:05.998602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.998550    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:35:05.998821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:05.998623    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:35:06.020172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:06.020104    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:06.416939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:06.416907    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"]
Oct 31 01:35:06.418213 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:06.418184    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710693b5_b3dc_4023_8f07_f19f1c2d4cff.slice/crio-e59d2cb61838fe424b3080cf7e9ebb3448730a290b67b1d6f85cd1fe40742b7c WatchSource:0}: Error finding container e59d2cb61838fe424b3080cf7e9ebb3448730a290b67b1d6f85cd1fe40742b7c: Status 404 returned error can't find the container with id e59d2cb61838fe424b3080cf7e9ebb3448730a290b67b1d6f85cd1fe40742b7c
Oct 31 01:35:06.712557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:06.712463    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw" event={"ID":"710693b5-b3dc-4023-8f07-f19f1c2d4cff","Type":"ContainerStarted","Data":"e59d2cb61838fe424b3080cf7e9ebb3448730a290b67b1d6f85cd1fe40742b7c"}
Oct 31 01:35:07.004998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:07.004963    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:35:07.005218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:07.005019    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:35:07.720864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:07.720830    5419 generic.go:356] "Generic (PLEG): container finished" podID="710693b5-b3dc-4023-8f07-f19f1c2d4cff" containerID="3f5d99168f1bd6138fd485ee8875e19cf9054c8d383a1f01db48a3b0144dd1eb" exitCode=0
Oct 31 01:35:07.721350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:07.720917    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw" event={"ID":"710693b5-b3dc-4023-8f07-f19f1c2d4cff","Type":"ContainerDied","Data":"3f5d99168f1bd6138fd485ee8875e19cf9054c8d383a1f01db48a3b0144dd1eb"}
Oct 31 01:35:08.002359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.002320    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:35:08.002567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.002386    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:35:08.325490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.325415    5419 patch_prober.go:29] interesting pod/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:35:08.325490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.325476    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:35:08.600965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.600877    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:35:08.906489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.906462    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:08.997554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.997516    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:35:08.998799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:08.998768    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:35:09.029379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.029350    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd\" (UniqueName: \"kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd\") pod \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") "
Oct 31 01:35:09.029652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.029630    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca\") pod \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") "
Oct 31 01:35:09.029706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.029672    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca\") pod \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\" (UID: \"710693b5-b3dc-4023-8f07-f19f1c2d4cff\") "
Oct 31 01:35:09.030330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.030305    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca" pod "710693b5-b3dc-4023-8f07-f19f1c2d4cff" (UID: "710693b5-b3dc-4023-8f07-f19f1c2d4cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:09.032856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.031908    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd" pod "710693b5-b3dc-4023-8f07-f19f1c2d4cff" (UID: "710693b5-b3dc-4023-8f07-f19f1c2d4cff"). InnerVolumeSpecName "kube-api-access-c8vrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:09.032856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.032169    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca" pod "710693b5-b3dc-4023-8f07-f19f1c2d4cff" (UID: "710693b5-b3dc-4023-8f07-f19f1c2d4cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:35:09.131284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.131204    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:09.131284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.131230    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/710693b5-b3dc-4023-8f07-f19f1c2d4cff-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:09.131284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.131243    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vrd\" (UniqueName: \"kubernetes.io/projected/710693b5-b3dc-4023-8f07-f19f1c2d4cff-kube-api-access-c8vrd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:09.736480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.736449    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"
Oct 31 01:35:09.736480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.736471    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw" event={"ID":"710693b5-b3dc-4023-8f07-f19f1c2d4cff","Type":"ContainerDied","Data":"e59d2cb61838fe424b3080cf7e9ebb3448730a290b67b1d6f85cd1fe40742b7c"}
Oct 31 01:35:09.736748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:09.736518    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59d2cb61838fe424b3080cf7e9ebb3448730a290b67b1d6f85cd1fe40742b7c"
Oct 31 01:35:10.010294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.006899    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:35:10.010294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.006959    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 31 01:35:10.112654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.112524    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:35:10.148592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148558    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.148783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148606    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.148783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148667    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.148783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148691    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.148954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148843    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:10.148954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148880    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.148954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148928    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.148954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148952    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.149126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148956    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:10.149126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148964    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:10.149126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.148977    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.149126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.149050    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:10.149126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.149059    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy\") pod \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\" (UID: \"1503f6f8-f1f6-4b40-a806-0657fc0da67b\") "
Oct 31 01:35:10.149981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.149955    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:10.150215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.150169    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.150215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.150209    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.150389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.150224    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.150389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.150238    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.150389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.150252    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.151520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.151496    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:35:10.151665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.151520    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:10.151868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.151845    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "kube-api-access-6kkpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:10.151976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.151874    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token" pod "1503f6f8-f1f6-4b40-a806-0657fc0da67b" (UID: "1503f6f8-f1f6-4b40-a806-0657fc0da67b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:10.250929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.250897    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.250929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.250925    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.250929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.250934    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1503f6f8-f1f6-4b40-a806-0657fc0da67b-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.251175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.250944    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kkpq\" (UniqueName: \"kubernetes.io/projected/1503f6f8-f1f6-4b40-a806-0657fc0da67b-kube-api-access-6kkpq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:10.768182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.762356    5419 generic.go:356] "Generic (PLEG): container finished" podID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerID="073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674" exitCode=0
Oct 31 01:35:10.768182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.762482    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" event={"ID":"1503f6f8-f1f6-4b40-a806-0657fc0da67b","Type":"ContainerDied","Data":"073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674"}
Oct 31 01:35:10.768182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.762574    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv" event={"ID":"1503f6f8-f1f6-4b40-a806-0657fc0da67b","Type":"ContainerDied","Data":"4c5de83293c168174b6d15ee64c110ddaac4c9b40822aff8dafeb6d6c3492e39"}
Oct 31 01:35:10.768182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.762606    5419 scope.go:122] "RemoveContainer" containerID="073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674"
Oct 31 01:35:10.768182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.762773    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"
Oct 31 01:35:10.800982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.800953    5419 scope.go:122] "RemoveContainer" containerID="85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"
Oct 31 01:35:10.810511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.810482    5419 scope.go:122] "RemoveContainer" containerID="073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674"
Oct 31 01:35:10.810783 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:10.810764    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674\": container with ID starting with 073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674 not found: ID does not exist" containerID="073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674"
Oct 31 01:35:10.810847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.810793    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674"} err="failed to get container status \"073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674\": rpc error: code = NotFound desc = could not find container \"073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674\": container with ID starting with 073f321b475cdeeba83ef727af20f603ba496141b661b0e164e4ba36e4f2b674 not found: ID does not exist"
Oct 31 01:35:10.810847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.810813    5419 scope.go:122] "RemoveContainer" containerID="85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"
Oct 31 01:35:10.811032 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:10.811010    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be\": container with ID starting with 85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be not found: ID does not exist" containerID="85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"
Oct 31 01:35:10.811069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.811040    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be"} err="failed to get container status \"85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be\": rpc error: code = NotFound desc = could not find container \"85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be\": container with ID starting with 85a2f5a4e6436215b33ecea016594e6245d4ab25a8b4a8f5520e0016e39141be not found: ID does not exist"
Oct 31 01:35:10.953891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:10.953857    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"]
Oct 31 01:35:11.004651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.004609    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:35:11.005373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.005344    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:35:11.006411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.006390    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 01:35:11.012712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.012684    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c70e7-openshift-default-ccfc4b465-gc8xv"]
Oct 31 01:35:11.106490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.106401    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"]
Oct 31 01:35:11.106669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.106608    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy" containerID="cri-o://3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c" gracePeriod=30
Oct 31 01:35:11.410376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.410302    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"]
Oct 31 01:35:11.410526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:11.410502    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" containerID="cri-o://31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721" gracePeriod=30
Oct 31 01:35:12.665320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.665287    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"]
Oct 31 01:35:12.666039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666017    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666041    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666079    5419 cpu_manager.go:414] "containerMap: removing container" podUID="710693b5-b3dc-4023-8f07-f19f1c2d4cff" containerName="revoke-keys"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666087    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710693b5-b3dc-4023-8f07-f19f1c2d4cff" containerName="revoke-keys"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666108    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666115    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666126    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy"
Oct 31 01:35:12.666171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666154    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy"
Oct 31 01:35:12.666527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666322    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e787433-9dc5-4e38-912f-b14630d9dda4" containerName="istio-proxy"
Oct 31 01:35:12.666527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666340    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy"
Oct 31 01:35:12.666527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.666354    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="710693b5-b3dc-4023-8f07-f19f1c2d4cff" containerName="revoke-keys"
Oct 31 01:35:12.671212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.671191    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.724511    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.724623    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.724724    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.724765    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfld8\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.724883    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.724947    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.725035    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.725073    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.725627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.725158    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.828410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.828376    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.829174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.828813    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.829375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.829198    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.829577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.829559    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.829693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.829680    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfld8\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.829857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.829845    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.829976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.829964    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.830124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.830110    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.830292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.830264    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.830452 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.830428    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.831433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.831348    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.832242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.831692    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.833840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.833811    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.834223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.834168    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.862051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.861840    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3d3023-openshift-default-dockercfg-dd5fx\""
Oct 31 01:35:12.912191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.898790    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.955182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.955050    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" path="/var/lib/kubelet/pods/1503f6f8-f1f6-4b40-a806-0657fc0da67b/volumes"
Oct 31 01:35:12.956422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.956285    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"]
Oct 31 01:35:12.956422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.956345    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.956750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.956697    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfld8\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.957787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.957764    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token\") pod \"e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") " pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:12.989062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:12.989029    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:13.466130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.466105    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"]
Oct 31 01:35:13.467682 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:13.467657    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d59487d_4f4d_40ca_98e3_37b0c7fcfc8a.slice/crio-7df0d12ac52504ec05a18b45e5e02b4d9cf138f0a02aed53ea194217f75ebfd9 WatchSource:0}: Error finding container 7df0d12ac52504ec05a18b45e5e02b4d9cf138f0a02aed53ea194217f75ebfd9: Status 404 returned error can't find the container with id 7df0d12ac52504ec05a18b45e5e02b4d9cf138f0a02aed53ea194217f75ebfd9
Oct 31 01:35:13.470207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.470178    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:13.470258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.470244    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:13.470306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.470274    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:13.885206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.885173    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" event={"ID":"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a","Type":"ContainerStarted","Data":"ee47b1509e454c68964592b36e2a114af6f878519b0be58871c4fdad09695435"}
Oct 31 01:35:13.885641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.885225    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" event={"ID":"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a","Type":"ContainerStarted","Data":"7df0d12ac52504ec05a18b45e5e02b4d9cf138f0a02aed53ea194217f75ebfd9"}
Oct 31 01:35:13.989657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:13.989629    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:14.582080 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:14.582032    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:35:14.995511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:14.995483    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:15.163087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:15.163035    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podStartSLOduration=3.163020707 podStartE2EDuration="3.163020707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:14.051503434 +0000 UTC m=+1835.352460011" watchObservedRunningTime="2026-10-31 01:35:15.163020707 +0000 UTC m=+1836.463977283"
Oct 31 01:35:15.898461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:15.898421    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:15.899268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:15.899250    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:35:16.373956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.373932    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:35:16.390094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390069    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390129    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390200    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390233    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390265    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390318    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390358    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.390485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390365    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390462    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390531    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert\") pod \"5073b593-a911-413c-8c9f-22226ee3e0f7\" (UID: \"5073b593-a911-413c-8c9f-22226ee3e0f7\") "
Oct 31 01:35:16.390629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390589    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.390723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390626    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.390723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.390656    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.391036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.391009    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:16.391190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.391035    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.391190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.391055    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.391190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.391070    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.391190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.391082    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.392589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.392565    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:16.392711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.392692    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:35:16.392951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.392931    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.393076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.392945    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc" pod "5073b593-a911-413c-8c9f-22226ee3e0f7" (UID: "5073b593-a911-413c-8c9f-22226ee3e0f7"). InnerVolumeSpecName "kube-api-access-xnlsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:16.492164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.492062    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.492164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.492096    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5073b593-a911-413c-8c9f-22226ee3e0f7-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.492164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.492106    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlsc\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-kube-api-access-xnlsc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.492164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.492119    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.492164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.492128    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5073b593-a911-413c-8c9f-22226ee3e0f7-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.659338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.659314    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:35:16.695157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695111    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695206    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695234    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695269    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695308    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695400    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695493    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.695568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695532    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695585    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.695687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695614    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695706    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket\") pod \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\" (UID: \"dd493a02-5f4e-4680-b77f-f4f29830a0e3\") "
Oct 31 01:35:16.695795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695753    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:16.695908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695813    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.695969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.695942    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.696526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.696504    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.696582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.696533    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.696582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.696549    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.696582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.696563    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.696582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.696577    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.697724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.697705    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:35:16.697823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.697797    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:16.698172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.698156    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:35:16.698232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.698170    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp" pod "dd493a02-5f4e-4680-b77f-f4f29830a0e3" (UID: "dd493a02-5f4e-4680-b77f-f4f29830a0e3"). InnerVolumeSpecName "kube-api-access-q8gmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:16.798263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.798195    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.798263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.798217    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.798263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.798227    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gmp\" (UniqueName: \"kubernetes.io/projected/dd493a02-5f4e-4680-b77f-f4f29830a0e3-kube-api-access-q8gmp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.798263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.798237    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd493a02-5f4e-4680-b77f-f4f29830a0e3-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:16.907994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.907962    5419 generic.go:356] "Generic (PLEG): container finished" podID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerID="3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c" exitCode=0
Oct 31 01:35:16.908220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.908015    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"
Oct 31 01:35:16.908220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.908048    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" event={"ID":"5073b593-a911-413c-8c9f-22226ee3e0f7","Type":"ContainerDied","Data":"3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c"}
Oct 31 01:35:16.908220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.908109    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w" event={"ID":"5073b593-a911-413c-8c9f-22226ee3e0f7","Type":"ContainerDied","Data":"cbc3bda2af37090ec21824a3cebb677178ac67b408b2043abe1ec3ec62c6cae0"}
Oct 31 01:35:16.908220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.908168    5419 scope.go:122] "RemoveContainer" containerID="3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c"
Oct 31 01:35:16.909568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.909538    5419 generic.go:356] "Generic (PLEG): container finished" podID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerID="31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721" exitCode=0
Oct 31 01:35:16.909702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.909568    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" event={"ID":"dd493a02-5f4e-4680-b77f-f4f29830a0e3","Type":"ContainerDied","Data":"31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721"}
Oct 31 01:35:16.909702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.909594    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"
Oct 31 01:35:16.909702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.909620    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" event={"ID":"dd493a02-5f4e-4680-b77f-f4f29830a0e3","Type":"ContainerDied","Data":"e8c11b78784879b03afec22f2cc0c02421f2ed757bf14366b9b1e1882e34c16f"}
Oct 31 01:35:16.919349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.919331    5419 scope.go:122] "RemoveContainer" containerID="752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"
Oct 31 01:35:16.934992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.934965    5419 scope.go:122] "RemoveContainer" containerID="3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c"
Oct 31 01:35:16.935303 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:16.935282    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c\": container with ID starting with 3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c not found: ID does not exist" containerID="3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c"
Oct 31 01:35:16.935367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.935314    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c"} err="failed to get container status \"3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c\": rpc error: code = NotFound desc = could not find container \"3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c\": container with ID starting with 3096a7af6eb470b102766c5b0ceba8f99887c7eb4a3db16ca9d24fbc973ca03c not found: ID does not exist"
Oct 31 01:35:16.935367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.935336    5419 scope.go:122] "RemoveContainer" containerID="752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"
Oct 31 01:35:16.935614 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:16.935595    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271\": container with ID starting with 752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271 not found: ID does not exist" containerID="752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"
Oct 31 01:35:16.935669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.935621    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271"} err="failed to get container status \"752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271\": rpc error: code = NotFound desc = could not find container \"752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271\": container with ID starting with 752049880c957192cfe100b144d6cea4b7b50bcef77a7ec6d90e9026466eb271 not found: ID does not exist"
Oct 31 01:35:16.935669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.935638    5419 scope.go:122] "RemoveContainer" containerID="31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721"
Oct 31 01:35:16.944502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.944480    5419 scope.go:122] "RemoveContainer" containerID="31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721"
Oct 31 01:35:16.944774 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:16.944753    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721\": container with ID starting with 31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721 not found: ID does not exist" containerID="31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721"
Oct 31 01:35:16.944832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:16.944780    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721"} err="failed to get container status \"31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721\": rpc error: code = NotFound desc = could not find container \"31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721\": container with ID starting with 31d38dd6aecad2be3ab94fa9cbca1b8214d88c89c0452f41acdbc228b824f721 not found: ID does not exist"
Oct 31 01:35:17.078281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.078175    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"]
Oct 31 01:35:17.080537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.080518    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-14ca09-openshift-default-659c788d6c-xhp9w"]
Oct 31 01:35:17.194213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.194176    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"]
Oct 31 01:35:17.199220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.199194    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r"]
Oct 31 01:35:17.328795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.328701    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"]
Oct 31 01:35:17.446922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.446892    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"]
Oct 31 01:35:17.640398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.640301    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:35:17.640398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:17.640373    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-2xh7r" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:35:18.880270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:18.877631    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" path="/var/lib/kubelet/pods/5073b593-a911-413c-8c9f-22226ee3e0f7/volumes"
Oct 31 01:35:18.880270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:18.878398    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" path="/var/lib/kubelet/pods/dd493a02-5f4e-4680-b77f-f4f29830a0e3/volumes"
Oct 31 01:35:22.329673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.329633    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj" podUID="4a978b4e-8e87-40e8-9e0e-28f014868bb5" containerName="payload-processing" containerID="cri-o://3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d" gracePeriod=25
Oct 31 01:35:22.447939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.447905    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg" podUID="96f51ada-f315-43cd-a9cc-7ca244b5c64d" containerName="payload-pre-processing" containerID="cri-o://d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849" gracePeriod=25
Oct 31 01:35:22.636353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.636326    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:35:22.697264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.697231    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t\" (UniqueName: \"kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t\") pod \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") "
Oct 31 01:35:22.697440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.697276    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume\") pod \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\" (UID: \"4a978b4e-8e87-40e8-9e0e-28f014868bb5\") "
Oct 31 01:35:22.697800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.697770    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume" pod "4a978b4e-8e87-40e8-9e0e-28f014868bb5" (UID: "4a978b4e-8e87-40e8-9e0e-28f014868bb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:22.699715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.699681    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t" pod "4a978b4e-8e87-40e8-9e0e-28f014868bb5" (UID: "4a978b4e-8e87-40e8-9e0e-28f014868bb5"). InnerVolumeSpecName "kube-api-access-j284t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:22.741692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.741666    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:35:22.798105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.798075    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume\") pod \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") "
Oct 31 01:35:22.798334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.798151    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6\" (UniqueName: \"kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6\") pod \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\" (UID: \"96f51ada-f315-43cd-a9cc-7ca244b5c64d\") "
Oct 31 01:35:22.798497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.798475    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume" pod "96f51ada-f315-43cd-a9cc-7ca244b5c64d" (UID: "96f51ada-f315-43cd-a9cc-7ca244b5c64d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:35:22.798697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.798682    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f51ada-f315-43cd-a9cc-7ca244b5c64d-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:22.798739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.798701    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j284t\" (UniqueName: \"kubernetes.io/projected/4a978b4e-8e87-40e8-9e0e-28f014868bb5-kube-api-access-j284t\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:22.798739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.798711    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a978b4e-8e87-40e8-9e0e-28f014868bb5-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:22.800392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.800374    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6" pod "96f51ada-f315-43cd-a9cc-7ca244b5c64d" (UID: "96f51ada-f315-43cd-a9cc-7ca244b5c64d"). InnerVolumeSpecName "kube-api-access-9swx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:35:22.900659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.900573    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swx6\" (UniqueName: \"kubernetes.io/projected/96f51ada-f315-43cd-a9cc-7ca244b5c64d-kube-api-access-9swx6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:35:22.992981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.992945    5419 generic.go:356] "Generic (PLEG): container finished" podID="4a978b4e-8e87-40e8-9e0e-28f014868bb5" containerID="3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d" exitCode=0
Oct 31 01:35:22.993212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.993006    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"
Oct 31 01:35:22.993212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.993026    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj" event={"ID":"4a978b4e-8e87-40e8-9e0e-28f014868bb5","Type":"ContainerDied","Data":"3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d"}
Oct 31 01:35:22.993212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.993101    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj" event={"ID":"4a978b4e-8e87-40e8-9e0e-28f014868bb5","Type":"ContainerDied","Data":"fdf52b447c0785b93a20337d896456896d43242dd99daf56e6b5fe59bf96de90"}
Oct 31 01:35:22.993212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.993126    5419 scope.go:122] "RemoveContainer" containerID="3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d"
Oct 31 01:35:22.995424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.994858    5419 generic.go:356] "Generic (PLEG): container finished" podID="96f51ada-f315-43cd-a9cc-7ca244b5c64d" containerID="d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849" exitCode=0
Oct 31 01:35:22.995424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.994914    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"
Oct 31 01:35:22.995424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.994938    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg" event={"ID":"96f51ada-f315-43cd-a9cc-7ca244b5c64d","Type":"ContainerDied","Data":"d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849"}
Oct 31 01:35:22.995424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:22.994974    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg" event={"ID":"96f51ada-f315-43cd-a9cc-7ca244b5c64d","Type":"ContainerDied","Data":"8e91c13c00dba1c9e266638fb8cde7fcd629138c45d8404bb6a197e72783d944"}
Oct 31 01:35:23.003417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.003400    5419 scope.go:122] "RemoveContainer" containerID="3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d"
Oct 31 01:35:23.003718 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:23.003699    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d\": container with ID starting with 3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d not found: ID does not exist" containerID="3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d"
Oct 31 01:35:23.003772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.003727    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d"} err="failed to get container status \"3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d\": rpc error: code = NotFound desc = could not find container \"3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d\": container with ID starting with 3b7da15483ed478466d07f76a8e55107f8c2bbe9f769d78b927d3c4177bcc97d not found: ID does not exist"
Oct 31 01:35:23.003772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.003744    5419 scope.go:122] "RemoveContainer" containerID="d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849"
Oct 31 01:35:23.012212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.012192    5419 scope.go:122] "RemoveContainer" containerID="d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849"
Oct 31 01:35:23.012486 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:23.012458    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849\": container with ID starting with d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849 not found: ID does not exist" containerID="d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849"
Oct 31 01:35:23.012549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.012493    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849"} err="failed to get container status \"d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849\": rpc error: code = NotFound desc = could not find container \"d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849\": container with ID starting with d3bb2a67c2de33803e1d2f06e68496ab984a0f777ccba823c8ad6ecdbd7be849 not found: ID does not exist"
Oct 31 01:35:23.155738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.155656    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"]
Oct 31 01:35:23.160372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.160338    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-57ea84-6898c65697-gwhtj"]
Oct 31 01:35:23.267910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.267870    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"]
Oct 31 01:35:23.270176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.270149    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-57ea84-85d5768fdc-p58qg"]
Oct 31 01:35:23.871974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.871934    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"]
Oct 31 01:35:23.872688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872664    5419 cpu_manager.go:414] "containerMap: removing container" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy"
Oct 31 01:35:23.872688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872689    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy"
Oct 31 01:35:23.872688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872715    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy"
Oct 31 01:35:23.872688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872723    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872752    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4a978b4e-8e87-40e8-9e0e-28f014868bb5" containerName="payload-processing"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872760    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a978b4e-8e87-40e8-9e0e-28f014868bb5" containerName="payload-processing"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872788    5419 cpu_manager.go:414] "containerMap: removing container" podUID="96f51ada-f315-43cd-a9cc-7ca244b5c64d" containerName="payload-pre-processing"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872796    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f51ada-f315-43cd-a9cc-7ca244b5c64d" containerName="payload-pre-processing"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872953    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd493a02-5f4e-4680-b77f-f4f29830a0e3" containerName="istio-proxy"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872967    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872981    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a978b4e-8e87-40e8-9e0e-28f014868bb5" containerName="payload-processing"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.872990    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1503f6f8-f1f6-4b40-a806-0657fc0da67b" containerName="istio-proxy"
Oct 31 01:35:23.873018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.873002    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f51ada-f315-43cd-a9cc-7ca244b5c64d" containerName="payload-pre-processing"
Oct 31 01:35:23.877780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.877756    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:23.917677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.917644    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtzp\" (UniqueName: \"kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:23.917830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.917762    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:23.929527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.929484    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3d3023-dockercfg-gx2cx\""
Oct 31 01:35:23.930305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.930282    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3d3023\""
Oct 31 01:35:23.932842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:23.932821    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"]
Oct 31 01:35:24.035802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.032083    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtzp\" (UniqueName: \"kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:24.035802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.032186    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:24.035802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.033089    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:24.096298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.091797    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"]
Oct 31 01:35:24.096298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.093019    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtzp\" (UniqueName: \"kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:24.110058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.106784    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy"
Oct 31 01:35:24.110058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.106820    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy"
Oct 31 01:35:24.110058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.107021    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5073b593-a911-413c-8c9f-22226ee3e0f7" containerName="istio-proxy"
Oct 31 01:35:24.117579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.114488    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.123541 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.123384    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"]
Oct 31 01:35:24.192017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.191985    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:24.235067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.235019    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.235067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.235073    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwf4p\" (UniqueName: \"kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p\") pod \"payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.336641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.336601    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.336814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.336661    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwf4p\" (UniqueName: \"kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p\") pod \"payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.337332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.337312    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.393285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.393211    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwf4p\" (UniqueName: \"kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p\") pod \"payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.442384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.442351    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:24.568660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.568636    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"]
Oct 31 01:35:24.571027 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:24.570995    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c03479_f974_415c_8610_8d73bcb7d9e3.slice/crio-d334ffed0e82fb20f1b60d1fdc9cfa3495300ef2a323abba96baffba898a488e WatchSource:0}: Error finding container d334ffed0e82fb20f1b60d1fdc9cfa3495300ef2a323abba96baffba898a488e: Status 404 returned error can't find the container with id d334ffed0e82fb20f1b60d1fdc9cfa3495300ef2a323abba96baffba898a488e
Oct 31 01:35:24.836856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.836824    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a978b4e-8e87-40e8-9e0e-28f014868bb5" path="/var/lib/kubelet/pods/4a978b4e-8e87-40e8-9e0e-28f014868bb5/volumes"
Oct 31 01:35:24.837260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.837247    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f51ada-f315-43cd-a9cc-7ca244b5c64d" path="/var/lib/kubelet/pods/96f51ada-f315-43cd-a9cc-7ca244b5c64d/volumes"
Oct 31 01:35:24.848080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:24.848061    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"]
Oct 31 01:35:24.849721 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:24.849690    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f82da23_0a4e_44d9_a0bd_0e876042b2e7.slice/crio-c6182fed2f092ebf59f5b74cc834be7c5553ecc913c0dc486a2990e961ed8b05 WatchSource:0}: Error finding container c6182fed2f092ebf59f5b74cc834be7c5553ecc913c0dc486a2990e961ed8b05: Status 404 returned error can't find the container with id c6182fed2f092ebf59f5b74cc834be7c5553ecc913c0dc486a2990e961ed8b05
Oct 31 01:35:25.105210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.105103    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn" event={"ID":"19c03479-f974-415c-8610-8d73bcb7d9e3","Type":"ContainerStarted","Data":"82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2"}
Oct 31 01:35:25.105210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.105182    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn" event={"ID":"19c03479-f974-415c-8610-8d73bcb7d9e3","Type":"ContainerStarted","Data":"d334ffed0e82fb20f1b60d1fdc9cfa3495300ef2a323abba96baffba898a488e"}
Oct 31 01:35:25.105772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.105324    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:25.106856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.106833    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2" event={"ID":"3f82da23-0a4e-44d9-a0bd-0e876042b2e7","Type":"ContainerStarted","Data":"9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4"}
Oct 31 01:35:25.106993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.106870    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2" event={"ID":"3f82da23-0a4e-44d9-a0bd-0e876042b2e7","Type":"ContainerStarted","Data":"c6182fed2f092ebf59f5b74cc834be7c5553ecc913c0dc486a2990e961ed8b05"}
Oct 31 01:35:25.106993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.106909    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:25.221183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.221105    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn" podStartSLOduration=2.221091236 podStartE2EDuration="2.221091236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:25.219055287 +0000 UTC m=+1846.520011864" watchObservedRunningTime="2026-10-31 01:35:25.221091236 +0000 UTC m=+1846.522047814"
Oct 31 01:35:25.358374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:25.353786    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2" podStartSLOduration=1.3537629930000001 podStartE2EDuration="1.353762993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:25.352027569 +0000 UTC m=+1846.652984159" watchObservedRunningTime="2026-10-31 01:35:25.353762993 +0000 UTC m=+1846.654719572"
Oct 31 01:35:36.117280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:36.117247    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:35:36.117792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:36.117327    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:35:37.951709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.951674    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"]
Oct 31 01:35:37.956714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.956682    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.997784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.997753    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.997970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.997818    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.997970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.997913    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8n2\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.998109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.997973    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.998109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.998014    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.998109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.998090    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.998319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.998160    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.998319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.998196    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:37.998319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:37.998281    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.008467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.008438    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-71bc98-openshift-default-dockercfg-dbm7d\""
Oct 31 01:35:38.060632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.060595    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"]
Oct 31 01:35:38.128163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.116221    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.129029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.128996    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.142496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.142462    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.145306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.145259    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8n2\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.145935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.145890    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.146664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.146521    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.146925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.146900    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.147617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.147597    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.148249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.148116    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.148560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.148539    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.149699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.148973    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.150097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.150034    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.150097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.149768    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.153288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.153265    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.154043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.153851    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.166941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.166911    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.202479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.202410    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8n2\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.203009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.202977    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token\") pod \"e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") " pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.272441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.271596    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:38.703405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.703372    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"]
Oct 31 01:35:38.740022 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:38.739988    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01660afd_9883_4a1b_9056_2885f954648b.slice/crio-f3ca3bee1227e5a4b667c8fd3345c89709d42ea08e40b789ea780b6215d05de4 WatchSource:0}: Error finding container f3ca3bee1227e5a4b667c8fd3345c89709d42ea08e40b789ea780b6215d05de4: Status 404 returned error can't find the container with id f3ca3bee1227e5a4b667c8fd3345c89709d42ea08e40b789ea780b6215d05de4
Oct 31 01:35:38.743130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.743096    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:38.743253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.743200    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:38.743318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:38.743249    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:35:39.103433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.103399    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:35:39.202717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.202682    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" event={"ID":"01660afd-9883-4a1b-9056-2885f954648b","Type":"ContainerStarted","Data":"dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"}
Oct 31 01:35:39.202927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.202734    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" event={"ID":"01660afd-9883-4a1b-9056-2885f954648b","Type":"ContainerStarted","Data":"f3ca3bee1227e5a4b667c8fd3345c89709d42ea08e40b789ea780b6215d05de4"}
Oct 31 01:35:39.272959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.272924    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:39.274455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.274431    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:35:39.274615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.274485    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 01:35:39.322643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:39.322591    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podStartSLOduration=2.322573784 podStartE2EDuration="2.322573784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:39.320422598 +0000 UTC m=+1860.621379186" watchObservedRunningTime="2026-10-31 01:35:39.322573784 +0000 UTC m=+1860.623530361"
Oct 31 01:35:40.070007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:40.069975    5419 scope.go:122] "RemoveContainer" containerID="69684c0660c111545130688d9f4fe519bf53c14c83cbd8054bfa69ebbe6fb652"
Oct 31 01:35:40.079705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:40.079686    5419 scope.go:122] "RemoveContainer" containerID="05f8b201587c1cac1bbd92209be59843526b473404896173f0ef9a171b864d9c"
Oct 31 01:35:40.276520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:40.276487    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:41.215229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:41.215196    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:41.216104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:41.216082    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:35:44.584080 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:35:44.584044    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:35:45.990663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:45.990632    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"]
Oct 31 01:35:45.996226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:45.996207    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.057163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.055376    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"]
Oct 31 01:35:46.105866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.105834    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-71bc98-dockercfg-9zgcl\""
Oct 31 01:35:46.106735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.106219    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-71bc98\""
Oct 31 01:35:46.134818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.134780    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr49h\" (UniqueName: \"kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h\") pod \"payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.140387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.135050    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"]
Oct 31 01:35:46.141042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.140789    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.144178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.143671    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.227667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.227007    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"]
Oct 31 01:35:46.241804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.241691    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.241804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.241780    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-585zq\" (UniqueName: \"kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq\") pod \"payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.242040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.241913    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr49h\" (UniqueName: \"kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h\") pod \"payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.242040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.242002    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.242938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.242913    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.298218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.298178    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr49h\" (UniqueName: \"kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h\") pod \"payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.309413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.309390    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:46.342989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.342953    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585zq\" (UniqueName: \"kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq\") pod \"payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.343173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.343150    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.343805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.343785    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.401433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.401402    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-585zq\" (UniqueName: \"kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq\") pod \"payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.541421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.541337    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:46.690697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.690670    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"]
Oct 31 01:35:46.692202 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:46.692172    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914cee20_683f_4545_bc7f_fb81b3813a06.slice/crio-4cda3e49fc7dc98515146d46876e574c09498c9fb5b8146cdcf708af426cbb6e WatchSource:0}: Error finding container 4cda3e49fc7dc98515146d46876e574c09498c9fb5b8146cdcf708af426cbb6e: Status 404 returned error can't find the container with id 4cda3e49fc7dc98515146d46876e574c09498c9fb5b8146cdcf708af426cbb6e
Oct 31 01:35:46.917037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:46.917007    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"]
Oct 31 01:35:46.918236 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:35:46.918204    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f66be6_fbda_413f_b547_299a6b16726e.slice/crio-78423caad306252b3ad12843abd42df55ef1143061290bf492221cd74fd8dbfe WatchSource:0}: Error finding container 78423caad306252b3ad12843abd42df55ef1143061290bf492221cd74fd8dbfe: Status 404 returned error can't find the container with id 78423caad306252b3ad12843abd42df55ef1143061290bf492221cd74fd8dbfe
Oct 31 01:35:47.255920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.255883    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf" event={"ID":"914cee20-683f-4545-bc7f-fb81b3813a06","Type":"ContainerStarted","Data":"76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f"}
Oct 31 01:35:47.256455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.255942    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf" event={"ID":"914cee20-683f-4545-bc7f-fb81b3813a06","Type":"ContainerStarted","Data":"4cda3e49fc7dc98515146d46876e574c09498c9fb5b8146cdcf708af426cbb6e"}
Oct 31 01:35:47.256455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.255962    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:47.257735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.257706    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh" event={"ID":"e9f66be6-fbda-413f-b547-299a6b16726e","Type":"ContainerStarted","Data":"08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f"}
Oct 31 01:35:47.257872 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.257756    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh" event={"ID":"e9f66be6-fbda-413f-b547-299a6b16726e","Type":"ContainerStarted","Data":"78423caad306252b3ad12843abd42df55ef1143061290bf492221cd74fd8dbfe"}
Oct 31 01:35:47.257872 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.257785    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:35:47.370258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.370205    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf" podStartSLOduration=2.370192324 podStartE2EDuration="2.370192324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:47.367957083 +0000 UTC m=+1868.668913660" watchObservedRunningTime="2026-10-31 01:35:47.370192324 +0000 UTC m=+1868.671148901"
Oct 31 01:35:47.493183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:47.491076    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh" podStartSLOduration=1.4910527120000001 podStartE2EDuration="1.491052712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:47.490501214 +0000 UTC m=+1868.791457803" watchObservedRunningTime="2026-10-31 01:35:47.491052712 +0000 UTC m=+1868.792009489"
Oct 31 01:35:58.267096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:58.267067    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 01:35:58.267539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:35:58.267132    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 01:36:14.583193 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:36:14.583131    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:36:21.312556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:21.312460    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"]
Oct 31 01:36:21.315587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:21.315557    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-bc2046-jm68d"]
Oct 31 01:36:21.347738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:21.347710    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:36:22.843936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:22.843900    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b133c1-b827-4d38-9845-01d98e723e9d" path="/var/lib/kubelet/pods/41b133c1-b827-4d38-9845-01d98e723e9d/volumes"
Oct 31 01:36:28.105744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:28.105709    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"]
Oct 31 01:36:28.106199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:28.105940    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy" containerID="cri-o://a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e" gracePeriod=30
Oct 31 01:36:29.833910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.833877    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"]
Oct 31 01:36:29.837845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.837820    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.892438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.891555    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-99c8bd-openshift-default-dockercfg-vppnj\""
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988682    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988734    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988794    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988849    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988874    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdb5\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988914    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.988964    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.989025    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:29.996200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:29.989199    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.016368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.016243    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"]
Oct 31 01:36:30.092053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.091124    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.092876 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.092811    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.093058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.093036    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.093295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.093276    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.095473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.095454    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.095670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.095653    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdb5\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.096273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.096252    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.100180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.096678    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.100354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.100333    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.101279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.101099    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.101544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.101524    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.101976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.101880    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.102289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.102271    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.102744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.102619    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.103681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.103640    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.116431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.116387    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.151773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.151683    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.155275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.155244    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdb5\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5\") pod \"e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") " pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.157441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.157213    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:30.615386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.615360    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"]
Oct 31 01:36:30.616346 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:36:30.616316    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5b7129_09bc_45e9_b97a_5759d316f60f.slice/crio-52b0a523f4ba196235d94f52058a56e25490b0860fe71802f50a0a0a63b7252c WatchSource:0}: Error finding container 52b0a523f4ba196235d94f52058a56e25490b0860fe71802f50a0a0a63b7252c: Status 404 returned error can't find the container with id 52b0a523f4ba196235d94f52058a56e25490b0860fe71802f50a0a0a63b7252c
Oct 31 01:36:30.619064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.619031    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:30.619191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.619111    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:30.619266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:30.619195    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:31.601132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:31.601096    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" event={"ID":"aa5b7129-09bc-45e9-b97a-5759d316f60f","Type":"ContainerStarted","Data":"e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"}
Oct 31 01:36:31.601584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:31.601178    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" event={"ID":"aa5b7129-09bc-45e9-b97a-5759d316f60f","Type":"ContainerStarted","Data":"52b0a523f4ba196235d94f52058a56e25490b0860fe71802f50a0a0a63b7252c"}
Oct 31 01:36:31.816387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:31.816326    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podStartSLOduration=2.816307612 podStartE2EDuration="2.816307612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:31.766251119 +0000 UTC m=+1913.067207721" watchObservedRunningTime="2026-10-31 01:36:31.816307612 +0000 UTC m=+1913.117264190"
Oct 31 01:36:31.943355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:31.943267    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:36:31.951668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:31.951623    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:36:32.158346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.158312    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:32.164828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.164798    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:32.609260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.609229    5419 generic.go:356] "Generic (PLEG): container finished" podID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerID="ee47b1509e454c68964592b36e2a114af6f878519b0be58871c4fdad09695435" exitCode=0
Oct 31 01:36:32.609747 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.609307    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" event={"ID":"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a","Type":"ContainerDied","Data":"ee47b1509e454c68964592b36e2a114af6f878519b0be58871c4fdad09695435"}
Oct 31 01:36:32.609747 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.609688    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:32.609854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.609789    5419 scope.go:122] "RemoveContainer" containerID="ee47b1509e454c68964592b36e2a114af6f878519b0be58871c4fdad09695435"
Oct 31 01:36:32.610833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.610519    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:36:32.611392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.611370    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:32.611442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.611432    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:32.611491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:32.611462    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:33.431338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.431308    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:36:33.572576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572540    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.572778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572596    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.572778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572656    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.572778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572745    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.572947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572783    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.572947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572821    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.572947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572903    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.573109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.572958    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.573109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573093    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:33.573274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573156    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket\") pod \"e4228b72-6d15-46f9-a069-c68007a190a9\" (UID: \"e4228b72-6d15-46f9-a069-c68007a190a9\") "
Oct 31 01:36:33.573604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573579    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:33.574104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573713    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4228b72-6d15-46f9-a069-c68007a190a9-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.574104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573741    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:33.574323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573899    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:33.574323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.573937    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:33.605763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.605710    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:33.606473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.606431    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:33.607193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.607124    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:33.607412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.607389    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg" pod "e4228b72-6d15-46f9-a069-c68007a190a9" (UID: "e4228b72-6d15-46f9-a069-c68007a190a9"). InnerVolumeSpecName "kube-api-access-6n5tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:33.621007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.620966    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" event={"ID":"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a","Type":"ContainerStarted","Data":"ef1991dbd485668de006b8f549cf3cfe9558c91cfebe837582a83d7313468672"}
Oct 31 01:36:33.623188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.623163    5419 generic.go:356] "Generic (PLEG): container finished" podID="e4228b72-6d15-46f9-a069-c68007a190a9" containerID="a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e" exitCode=0
Oct 31 01:36:33.623307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.623224    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"
Oct 31 01:36:33.623307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.623256    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" event={"ID":"e4228b72-6d15-46f9-a069-c68007a190a9","Type":"ContainerDied","Data":"a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e"}
Oct 31 01:36:33.623431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.623321    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6" event={"ID":"e4228b72-6d15-46f9-a069-c68007a190a9","Type":"ContainerDied","Data":"331adc4e3fbea7943c77dbb9403c354c4bfe792afa7f690952d325496ce93a9b"}
Oct 31 01:36:33.623431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.623344    5419 scope.go:122] "RemoveContainer" containerID="a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e"
Oct 31 01:36:33.633741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.633720    5419 scope.go:122] "RemoveContainer" containerID="47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"
Oct 31 01:36:33.643288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.643264    5419 scope.go:122] "RemoveContainer" containerID="a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e"
Oct 31 01:36:33.643568 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:36:33.643547    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e\": container with ID starting with a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e not found: ID does not exist" containerID="a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e"
Oct 31 01:36:33.643630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.643578    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e"} err="failed to get container status \"a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e\": rpc error: code = NotFound desc = could not find container \"a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e\": container with ID starting with a83afdd85f222a84ba515559e0bfa8d749941458bd40c9bfe262789e19e8f11e not found: ID does not exist"
Oct 31 01:36:33.643630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.643607    5419 scope.go:122] "RemoveContainer" containerID="47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"
Oct 31 01:36:33.643824 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:36:33.643809    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f\": container with ID starting with 47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f not found: ID does not exist" containerID="47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"
Oct 31 01:36:33.643865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.643828    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f"} err="failed to get container status \"47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f\": rpc error: code = NotFound desc = could not find container \"47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f\": container with ID starting with 47c1b44b4e12f235755bdfc17bf28967f62f23cf0673d2e06fb7a45875dcb75f not found: ID does not exist"
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.699924    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.699956    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.699971    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.699983    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.699994    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n5tg\" (UniqueName: \"kubernetes.io/projected/e4228b72-6d15-46f9-a069-c68007a190a9-kube-api-access-6n5tg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.700004    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.700011    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4228b72-6d15-46f9-a069-c68007a190a9-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.700019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.700022    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4228b72-6d15-46f9-a069-c68007a190a9-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:33.855452 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.855422    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"]
Oct 31 01:36:33.862785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.862761    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bc2046-openshift-default-577d58fbfc-mf7p6"]
Oct 31 01:36:33.995183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.995075    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:36:33.999218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.998228    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 31 01:36:33.999218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:33.998296    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 31 01:36:34.837029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:34.836999    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" path="/var/lib/kubelet/pods/e4228b72-6d15-46f9-a069-c68007a190a9/volumes"
Oct 31 01:36:34.999632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:34.999588    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 31 01:36:34.999812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:34.999658    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 31 01:36:36.005360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:36.005325    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 31 01:36:36.006584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:36.006342    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 31 01:36:37.007163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:36.994627    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 31 01:36:37.007163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:36.994688    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 31 01:36:37.991163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:37.990166    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Oct 31 01:36:37.991163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:37.990237    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Oct 31 01:36:38.912224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:38.912189    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"]
Oct 31 01:36:38.959958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:38.959924    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57ea84-79fpw"]
Oct 31 01:36:38.993265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:38.993238    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:36:38.993615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:38.993595    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:36:38.994222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:38.994205    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:36:40.907425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:40.907393    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710693b5-b3dc-4023-8f07-f19f1c2d4cff" path="/var/lib/kubelet/pods/710693b5-b3dc-4023-8f07-f19f1c2d4cff/volumes"
Oct 31 01:36:44.609435 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:36:44.609396    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:36:45.660109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:45.660070    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"]
Oct 31 01:36:45.660598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:45.660367    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" containerID="cri-o://f3c9916f17e7fcfbc0f9980d947c02ff9bf5e442648a477069386577801a8b03" gracePeriod=30
Oct 31 01:36:49.300827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:49.300747    5419 patch_prober.go:29] interesting pod/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:36:49.301388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:49.300835    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:36:50.796581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:50.796549    5419 generic.go:356] "Generic (PLEG): container finished" podID="1f200f95-279c-4bde-8336-d3975396e88b" containerID="f3c9916f17e7fcfbc0f9980d947c02ff9bf5e442648a477069386577801a8b03" exitCode=0
Oct 31 01:36:50.796991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:50.796639    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" event={"ID":"1f200f95-279c-4bde-8336-d3975396e88b","Type":"ContainerDied","Data":"f3c9916f17e7fcfbc0f9980d947c02ff9bf5e442648a477069386577801a8b03"}
Oct 31 01:36:50.796991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:50.796716    5419 scope.go:122] "RemoveContainer" containerID="d5ec4f1e5cbe88e7e69ba5a8505ef85d81a139b89067443ae0c847a6f34bfc3e"
Oct 31 01:36:50.934979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:50.934922    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:36:51.033543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033509    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033570    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033634    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033674    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033762    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033801    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033846    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.033927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033911    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.034132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033927    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:51.034132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.033949    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs\") pod \"1f200f95-279c-4bde-8336-d3975396e88b\" (UID: \"1f200f95-279c-4bde-8336-d3975396e88b\") "
Oct 31 01:36:51.034283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034206    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:51.034283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034229    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:51.034587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034441    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:51.034746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034667    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:51.034746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034721    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f200f95-279c-4bde-8336-d3975396e88b-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.034746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034736    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.034746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034746    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.034954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.034755    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.036809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.036763    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:51.036809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.036786    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "kube-api-access-nd4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:51.037264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.037223    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:51.037565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.037548    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo" pod "1f200f95-279c-4bde-8336-d3975396e88b" (UID: "1f200f95-279c-4bde-8336-d3975396e88b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:51.136269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.136231    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.136269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.136272    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.136479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.136289    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4hq\" (UniqueName: \"kubernetes.io/projected/1f200f95-279c-4bde-8336-d3975396e88b-kube-api-access-nd4hq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.136479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.136304    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f200f95-279c-4bde-8336-d3975396e88b-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.136479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.136317    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f200f95-279c-4bde-8336-d3975396e88b-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:51.804401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.804370    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"
Oct 31 01:36:51.804885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.804409    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp" event={"ID":"1f200f95-279c-4bde-8336-d3975396e88b","Type":"ContainerDied","Data":"3a3fea3a46996dd4240d7c1fc837b1ac4f21d25f10b144c5ac6c8cccb93aa673"}
Oct 31 01:36:51.804885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.804489    5419 scope.go:122] "RemoveContainer" containerID="f3c9916f17e7fcfbc0f9980d947c02ff9bf5e442648a477069386577801a8b03"
Oct 31 01:36:51.806230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.806209    5419 generic.go:356] "Generic (PLEG): container finished" podID="01660afd-9883-4a1b-9056-2885f954648b" containerID="dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451" exitCode=0
Oct 31 01:36:51.806344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.806249    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" event={"ID":"01660afd-9883-4a1b-9056-2885f954648b","Type":"ContainerDied","Data":"dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"}
Oct 31 01:36:51.806657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.806643    5419 scope.go:122] "RemoveContainer" containerID="dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"
Oct 31 01:36:51.808180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.808132    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:51.808258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.808223    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:51.808300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:51.808258    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:36:52.020262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:52.020187    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"]
Oct 31 01:36:52.069438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:52.069395    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-57ea84-openshift-default-5d444895ff-8nrrp"]
Oct 31 01:36:52.815727 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:52.815691    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" event={"ID":"01660afd-9883-4a1b-9056-2885f954648b","Type":"ContainerStarted","Data":"92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982"}
Oct 31 01:36:52.837240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:52.837216    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f200f95-279c-4bde-8336-d3975396e88b" path="/var/lib/kubelet/pods/1f200f95-279c-4bde-8336-d3975396e88b/volumes"
Oct 31 01:36:53.045994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:53.045960    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"]
Oct 31 01:36:53.046224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:53.046201    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" containerID="cri-o://670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601" gracePeriod=30
Oct 31 01:36:53.272208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:53.272132    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:36:53.273423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:53.273396    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:36:53.273545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:53.273447    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 01:36:54.272350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:54.272315    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:36:54.272744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:54.272370    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 01:36:55.272575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.272540    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:36:55.272984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.272600    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 01:36:55.643497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.643411    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"]
Oct 31 01:36:55.644072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644049    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy"
Oct 31 01:36:55.644235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644074    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy"
Oct 31 01:36:55.644235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644103    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy"
Oct 31 01:36:55.644235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644111    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy"
Oct 31 01:36:55.644235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644162    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy"
Oct 31 01:36:55.644235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644171    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy"
Oct 31 01:36:55.644511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644343    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy"
Oct 31 01:36:55.644511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644362    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy"
Oct 31 01:36:55.644511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644375    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy"
Oct 31 01:36:55.644511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.644385    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4228b72-6d15-46f9-a069-c68007a190a9" containerName="istio-proxy"
Oct 31 01:36:55.647715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.647693    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.741161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.730968    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2wf\" (UniqueName: \"kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf\") pod \"payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.741161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.731061    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.753456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.753419    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-53af08-dockercfg-gxpz2\""
Oct 31 01:36:55.769006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.768355    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-53af08\""
Oct 31 01:36:55.820162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.770659    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"]
Oct 31 01:36:55.835235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.835198    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2wf\" (UniqueName: \"kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf\") pod \"payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.836349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.836320    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.837782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.837750    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.867752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.867710    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"]
Oct 31 01:36:55.868991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.868964    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy"
Oct 31 01:36:55.868991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.868988    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f200f95-279c-4bde-8336-d3975396e88b" containerName="istio-proxy"
Oct 31 01:36:55.873338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.873314    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:55.931518 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.931468    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"]
Oct 31 01:36:55.938813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.938784    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpkp\" (UniqueName: \"kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp\") pod \"payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:55.939080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.939064    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:55.942698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.942670    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2wf\" (UniqueName: \"kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf\") pod \"payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:55.961355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:55.961335    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:56.040556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.040523    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:56.040703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.040687    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpkp\" (UniqueName: \"kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp\") pod \"payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:56.041339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.041321    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:56.103662 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.103591    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpkp\" (UniqueName: \"kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp\") pod \"payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:56.215216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.215117    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:36:56.249981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.249954    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:56.273063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.273032    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:36:56.273421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.273106    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 01:36:56.399544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.399513    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"]
Oct 31 01:36:56.410185 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:36:56.404687    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29194385_841c_4ed1_b17f_f765408fd255.slice/crio-92ba322a98b9688e17b88fbd97cfa7a9834ad41c40fc64382951e40c590bcd86 WatchSource:0}: Error finding container 92ba322a98b9688e17b88fbd97cfa7a9834ad41c40fc64382951e40c590bcd86: Status 404 returned error can't find the container with id 92ba322a98b9688e17b88fbd97cfa7a9834ad41c40fc64382951e40c590bcd86
Oct 31 01:36:56.650099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.650078    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"]
Oct 31 01:36:56.652583 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:36:56.652539    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e37029e_a52f_49be_9447_80d93268c132.slice/crio-c2dd7ae5ca141d08d8d3c229bdd69a72e6ca893be91a85005a1f56cad72c043e WatchSource:0}: Error finding container c2dd7ae5ca141d08d8d3c229bdd69a72e6ca893be91a85005a1f56cad72c043e: Status 404 returned error can't find the container with id c2dd7ae5ca141d08d8d3c229bdd69a72e6ca893be91a85005a1f56cad72c043e
Oct 31 01:36:56.850188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.850150    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk" event={"ID":"1e37029e-a52f-49be-9447-80d93268c132","Type":"ContainerStarted","Data":"cabeb97ff65fb4df92094304d032060d725e1ae3c433f2bc7b87b61d94b651b3"}
Oct 31 01:36:56.850362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.850204    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk" event={"ID":"1e37029e-a52f-49be-9447-80d93268c132","Type":"ContainerStarted","Data":"c2dd7ae5ca141d08d8d3c229bdd69a72e6ca893be91a85005a1f56cad72c043e"}
Oct 31 01:36:56.850362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.850356    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:36:56.852099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.852075    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz" event={"ID":"29194385-841c-4ed1-b17f-f765408fd255","Type":"ContainerStarted","Data":"f5cbced8ec0ebdb4b88b822f0e7dd863d1ea986aac1e62983b7feafe8e23a014"}
Oct 31 01:36:56.852233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.852110    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz" event={"ID":"29194385-841c-4ed1-b17f-f765408fd255","Type":"ContainerStarted","Data":"92ba322a98b9688e17b88fbd97cfa7a9834ad41c40fc64382951e40c590bcd86"}
Oct 31 01:36:56.852233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.852131    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:36:56.964907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:56.964853    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk" podStartSLOduration=1.96483958 podStartE2EDuration="1.96483958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:56.96399188 +0000 UTC m=+1938.264948457" watchObservedRunningTime="2026-10-31 01:36:56.96483958 +0000 UTC m=+1938.265796162"
Oct 31 01:36:57.103217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:57.100068    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz" podStartSLOduration=2.100049943 podStartE2EDuration="2.100049943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:57.09662719 +0000 UTC m=+1938.397583767" watchObservedRunningTime="2026-10-31 01:36:57.100049943 +0000 UTC m=+1938.401006533"
Oct 31 01:36:57.272748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:57.272711    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Oct 31 01:36:57.272922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:57.272764    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Oct 31 01:36:57.700126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:57.700095    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:36:57.700678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:57.700213    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:36:58.276972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.276942    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:36:58.277988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.277964    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 01:36:58.308855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.308834    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:36:58.386451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386384    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386419    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386441    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386502    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386576    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386648    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386653    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:58.386958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386737    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386809    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386858    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc\") pod \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\" (UID: \"ae37bc44-518e-4d47-8dc9-a3a493656aaf\") "
Oct 31 01:36:58.386958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386876    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:58.387331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.386966    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:36:58.387331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387001    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:58.387331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387091    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:58.387536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387504    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.387536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387526    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.387654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387542    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.387654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387559    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.387654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.387575    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.388935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.388909    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:36:58.389442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.389411    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:36:58.389538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.389453    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:58.389538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.389483    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc" pod "ae37bc44-518e-4d47-8dc9-a3a493656aaf" (UID: "ae37bc44-518e-4d47-8dc9-a3a493656aaf"). InnerVolumeSpecName "kube-api-access-fvclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:36:58.488655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.488622    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.488655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.488646    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.488655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.488655    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae37bc44-518e-4d47-8dc9-a3a493656aaf-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.488850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.488664    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvclc\" (UniqueName: \"kubernetes.io/projected/ae37bc44-518e-4d47-8dc9-a3a493656aaf-kube-api-access-fvclc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:36:58.906310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.906270    5419 generic.go:356] "Generic (PLEG): container finished" podID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerID="670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601" exitCode=0
Oct 31 01:36:58.907331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.907298    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" event={"ID":"ae37bc44-518e-4d47-8dc9-a3a493656aaf","Type":"ContainerDied","Data":"670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601"}
Oct 31 01:36:58.907623 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.907588    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp" event={"ID":"ae37bc44-518e-4d47-8dc9-a3a493656aaf","Type":"ContainerDied","Data":"60a4ec9404802c3f5e1f81c4cc1c36f4410e36d629929f9cbdbfe57a12a1e799"}
Oct 31 01:36:58.907714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.907633    5419 scope.go:122] "RemoveContainer" containerID="670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601"
Oct 31 01:36:58.907764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.907335    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"
Oct 31 01:36:58.920753 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.920735    5419 scope.go:122] "RemoveContainer" containerID="670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601"
Oct 31 01:36:58.921058 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:36:58.921033    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601\": container with ID starting with 670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601 not found: ID does not exist" containerID="670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601"
Oct 31 01:36:58.921167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:58.921070    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601"} err="failed to get container status \"670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601\": rpc error: code = NotFound desc = could not find container \"670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601\": container with ID starting with 670ed909c13c35dc9c9459750ec7a9989870b24390f31496a96e3c152dcc6601 not found: ID does not exist"
Oct 31 01:36:59.071527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:59.071483    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"]
Oct 31 01:36:59.073811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:36:59.073791    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-nmdzp"]
Oct 31 01:37:00.844206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:00.844168    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" path="/var/lib/kubelet/pods/ae37bc44-518e-4d47-8dc9-a3a493656aaf/volumes"
Oct 31 01:37:03.932976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:03.932942    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"]
Oct 31 01:37:04.037630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.037589    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"]
Oct 31 01:37:04.275340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.275304    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"]
Oct 31 01:37:04.276403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.276375    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy"
Oct 31 01:37:04.276581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.276566    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy"
Oct 31 01:37:04.276888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.276869    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae37bc44-518e-4d47-8dc9-a3a493656aaf" containerName="istio-proxy"
Oct 31 01:37:04.282408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.281536    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.322999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.322933    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"]
Oct 31 01:37:04.333853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.333815    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9e5368-dockercfg-hkgjq\""
Oct 31 01:37:04.334087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.334068    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9e5368\""
Oct 31 01:37:04.401557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.401525    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.401717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.401655    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6z4\" (UniqueName: \"kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.503169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.503101    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6z4\" (UniqueName: \"kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.503381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.503321    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.504344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.504310    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.562273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.562210    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6z4\" (UniqueName: \"kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4\") pod \"payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:04.630479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:04.630442    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:05.005219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.005197    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"]
Oct 31 01:37:05.007485 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:37:05.007457    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a860479_c2cd_4a00_931d_0be8a188d7c1.slice/crio-0491bd2a1f5828d75aa659b87043265123054ddf0116a9da4dac4f4eb20f1a08 WatchSource:0}: Error finding container 0491bd2a1f5828d75aa659b87043265123054ddf0116a9da4dac4f4eb20f1a08: Status 404 returned error can't find the container with id 0491bd2a1f5828d75aa659b87043265123054ddf0116a9da4dac4f4eb20f1a08
Oct 31 01:37:05.009455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.009439    5419 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:37:05.043756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.043729    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"]
Oct 31 01:37:05.048768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.048751    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.115407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.115381    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"]
Oct 31 01:37:05.210959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.210916    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.211173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.211086    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fl5n\" (UniqueName: \"kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n\") pod \"payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.312682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.312593    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.312842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.312690    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fl5n\" (UniqueName: \"kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n\") pod \"payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.313340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.313318    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.385206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.381611    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fl5n\" (UniqueName: \"kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n\") pod \"payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.664396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.664310    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:05.966201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:05.966112    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft" event={"ID":"1a860479-c2cd-4a00-931d-0be8a188d7c1","Type":"ContainerStarted","Data":"0491bd2a1f5828d75aa659b87043265123054ddf0116a9da4dac4f4eb20f1a08"}
Oct 31 01:37:06.123502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:06.123475    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"]
Oct 31 01:37:06.125046 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:37:06.125016    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17f2d3b_c29e_4303_9f50_5feea583bf69.slice/crio-4d68fae4d7b49788241f63f8174a1d5f3b2e4d7849894a4ac3dea5ab4d91feed WatchSource:0}: Error finding container 4d68fae4d7b49788241f63f8174a1d5f3b2e4d7849894a4ac3dea5ab4d91feed: Status 404 returned error can't find the container with id 4d68fae4d7b49788241f63f8174a1d5f3b2e4d7849894a4ac3dea5ab4d91feed
Oct 31 01:37:06.975553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:06.975512    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv" event={"ID":"c17f2d3b-c29e-4303-9f50-5feea583bf69","Type":"ContainerStarted","Data":"4d68fae4d7b49788241f63f8174a1d5f3b2e4d7849894a4ac3dea5ab4d91feed"}
Oct 31 01:37:07.179286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:07.179253    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:37:07.884046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:07.883974    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:37:07.884245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:07.884052    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:37:07.985630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:07.985593    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft" event={"ID":"1a860479-c2cd-4a00-931d-0be8a188d7c1","Type":"ContainerStarted","Data":"d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3"}
Oct 31 01:37:07.986232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:07.986198    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:08.934294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:08.934256    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2" podUID="3f82da23-0a4e-44d9-a0bd-0e876042b2e7" containerName="payload-processing" containerID="cri-o://9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4" gracePeriod=25
Oct 31 01:37:08.999002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:08.998967    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv" event={"ID":"c17f2d3b-c29e-4303-9f50-5feea583bf69","Type":"ContainerStarted","Data":"6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f"}
Oct 31 01:37:08.999214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:08.999172    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:09.038191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.038099    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn" podUID="19c03479-f974-415c-8610-8d73bcb7d9e3" containerName="payload-pre-processing" containerID="cri-o://82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2" gracePeriod=25
Oct 31 01:37:09.115889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.115828    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft" podStartSLOduration=2.606476341 podStartE2EDuration="5.115810507s" totalImagesPullingTime="2.509334166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:08.223674868 +0000 UTC m=+1949.524631442" watchObservedRunningTime="2026-10-31 01:37:09.115810507 +0000 UTC m=+1950.416767086"
Oct 31 01:37:09.116058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.115948    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv" podStartSLOduration=2.316451216 podStartE2EDuration="4.115939364s" totalImagesPullingTime="1.799488148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:09.114123109 +0000 UTC m=+1950.415079730" watchObservedRunningTime="2026-10-31 01:37:09.115939364 +0000 UTC m=+1950.416895943"
Oct 31 01:37:09.217298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.217269    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:37:09.344285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.344260    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:37:09.388833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.388799    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume\") pod \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") "
Oct 31 01:37:09.388980 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.388849    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p\" (UniqueName: \"kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p\") pod \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\" (UID: \"3f82da23-0a4e-44d9-a0bd-0e876042b2e7\") "
Oct 31 01:37:09.389256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.389231    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume" pod "3f82da23-0a4e-44d9-a0bd-0e876042b2e7" (UID: "3f82da23-0a4e-44d9-a0bd-0e876042b2e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:37:09.391087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.391069    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p" pod "3f82da23-0a4e-44d9-a0bd-0e876042b2e7" (UID: "3f82da23-0a4e-44d9-a0bd-0e876042b2e7"). InnerVolumeSpecName "kube-api-access-fwf4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:09.489873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.489843    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume\") pod \"19c03479-f974-415c-8610-8d73bcb7d9e3\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") "
Oct 31 01:37:09.490044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.490035    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp\" (UniqueName: \"kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp\") pod \"19c03479-f974-415c-8610-8d73bcb7d9e3\" (UID: \"19c03479-f974-415c-8610-8d73bcb7d9e3\") "
Oct 31 01:37:09.490265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.490235    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume" pod "19c03479-f974-415c-8610-8d73bcb7d9e3" (UID: "19c03479-f974-415c-8610-8d73bcb7d9e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:37:09.490460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.490442    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:09.490557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.490462    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwf4p\" (UniqueName: \"kubernetes.io/projected/3f82da23-0a4e-44d9-a0bd-0e876042b2e7-kube-api-access-fwf4p\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:09.490557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.490472    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c03479-f974-415c-8610-8d73bcb7d9e3-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:09.492389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.492366    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp" pod "19c03479-f974-415c-8610-8d73bcb7d9e3" (UID: "19c03479-f974-415c-8610-8d73bcb7d9e3"). InnerVolumeSpecName "kube-api-access-kjtzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:37:09.591333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:09.591300    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtzp\" (UniqueName: \"kubernetes.io/projected/19c03479-f974-415c-8610-8d73bcb7d9e3-kube-api-access-kjtzp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:37:10.007115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.007080    5419 generic.go:356] "Generic (PLEG): container finished" podID="19c03479-f974-415c-8610-8d73bcb7d9e3" containerID="82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2" exitCode=0
Oct 31 01:37:10.007632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.007130    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"
Oct 31 01:37:10.007632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.007185    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn" event={"ID":"19c03479-f974-415c-8610-8d73bcb7d9e3","Type":"ContainerDied","Data":"82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2"}
Oct 31 01:37:10.007632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.007248    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn" event={"ID":"19c03479-f974-415c-8610-8d73bcb7d9e3","Type":"ContainerDied","Data":"d334ffed0e82fb20f1b60d1fdc9cfa3495300ef2a323abba96baffba898a488e"}
Oct 31 01:37:10.007632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.007273    5419 scope.go:122] "RemoveContainer" containerID="82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2"
Oct 31 01:37:10.009205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.009175    5419 generic.go:356] "Generic (PLEG): container finished" podID="3f82da23-0a4e-44d9-a0bd-0e876042b2e7" containerID="9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4" exitCode=0
Oct 31 01:37:10.009398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.009378    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"
Oct 31 01:37:10.009479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.009416    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2" event={"ID":"3f82da23-0a4e-44d9-a0bd-0e876042b2e7","Type":"ContainerDied","Data":"9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4"}
Oct 31 01:37:10.009479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.009461    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2" event={"ID":"3f82da23-0a4e-44d9-a0bd-0e876042b2e7","Type":"ContainerDied","Data":"c6182fed2f092ebf59f5b74cc834be7c5553ecc913c0dc486a2990e961ed8b05"}
Oct 31 01:37:10.018389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.018369    5419 scope.go:122] "RemoveContainer" containerID="82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2"
Oct 31 01:37:10.018664 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:37:10.018642    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2\": container with ID starting with 82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2 not found: ID does not exist" containerID="82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2"
Oct 31 01:37:10.018761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.018676    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2"} err="failed to get container status \"82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2\": rpc error: code = NotFound desc = could not find container \"82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2\": container with ID starting with 82b489d57c46d6a2933eca5067f19f4e2795b0f38f6503a01c225efbb12166a2 not found: ID does not exist"
Oct 31 01:37:10.018761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.018697    5419 scope.go:122] "RemoveContainer" containerID="9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4"
Oct 31 01:37:10.028880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.028864    5419 scope.go:122] "RemoveContainer" containerID="9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4"
Oct 31 01:37:10.029150 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:37:10.029117    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4\": container with ID starting with 9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4 not found: ID does not exist" containerID="9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4"
Oct 31 01:37:10.029221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.029161    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4"} err="failed to get container status \"9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4\": rpc error: code = NotFound desc = could not find container \"9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4\": container with ID starting with 9ba97cf5970e317e7a144c45621bd8e21aee643bf9cd43284ce074d90e63a1a4 not found: ID does not exist"
Oct 31 01:37:10.184590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.184554    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"]
Oct 31 01:37:10.234458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.234422    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-865744f895-8x2qn"]
Oct 31 01:37:10.297068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.297037    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"]
Oct 31 01:37:10.301111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.301084    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-78f9898c59-h98z2"]
Oct 31 01:37:10.838720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.838688    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c03479-f974-415c-8610-8d73bcb7d9e3" path="/var/lib/kubelet/pods/19c03479-f974-415c-8610-8d73bcb7d9e3/volumes"
Oct 31 01:37:10.839096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:10.839083    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f82da23-0a4e-44d9-a0bd-0e876042b2e7" path="/var/lib/kubelet/pods/3f82da23-0a4e-44d9-a0bd-0e876042b2e7/volumes"
Oct 31 01:37:14.583099 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:37:14.583061    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:37:16.046268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:16.046233    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"]
Oct 31 01:37:16.096924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:16.096887    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294520ea-nsmkl"]
Oct 31 01:37:16.837783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:16.837754    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbf8154-0759-4f2a-b8ca-f2085452d83a" path="/var/lib/kubelet/pods/dcbf8154-0759-4f2a-b8ca-f2085452d83a/volumes"
Oct 31 01:37:20.012308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:20.012279    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:37:20.012812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:20.012358    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:37:26.387476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.387440    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"]
Oct 31 01:37:26.388040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.388021    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3f82da23-0a4e-44d9-a0bd-0e876042b2e7" containerName="payload-processing"
Oct 31 01:37:26.388129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.388044    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f82da23-0a4e-44d9-a0bd-0e876042b2e7" containerName="payload-processing"
Oct 31 01:37:26.388129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.388104    5419 cpu_manager.go:414] "containerMap: removing container" podUID="19c03479-f974-415c-8610-8d73bcb7d9e3" containerName="payload-pre-processing"
Oct 31 01:37:26.388129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.388111    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c03479-f974-415c-8610-8d73bcb7d9e3" containerName="payload-pre-processing"
Oct 31 01:37:26.388304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.388244    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f82da23-0a4e-44d9-a0bd-0e876042b2e7" containerName="payload-processing"
Oct 31 01:37:26.388304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.388259    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c03479-f974-415c-8610-8d73bcb7d9e3" containerName="payload-pre-processing"
Oct 31 01:37:26.393229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.393212    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.580167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.577988    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-447154-openshift-default-dockercfg-9922r\""
Oct 31 01:37:26.641328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.641251    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"]
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643500    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643652    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643680    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643723    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643787    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643851    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7s7k\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643885    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643936    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.645175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.643970    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745537    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745575    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745598    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745634    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745668    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7s7k\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745687    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745714    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745742    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.745830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.745810    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.746278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.746004    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.746278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.746064    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.746278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.746185    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.746278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.746196    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.746419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.746350    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.748343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.748323    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.748556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.748539    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.828015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.827978    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7s7k\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:26.828278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:26.828044    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token\") pod \"e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") " pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:27.006458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:27.006424    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:27.442728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:27.439354    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"]
Oct 31 01:37:27.444662 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:37:27.444629    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf8aa23_f61d_475f_9752_80e896ec3eab.slice/crio-2b5b0cc052e03d84b7787d33882cf72e1f15156dcfcea724a0cf0b02276a1966 WatchSource:0}: Error finding container 2b5b0cc052e03d84b7787d33882cf72e1f15156dcfcea724a0cf0b02276a1966: Status 404 returned error can't find the container with id 2b5b0cc052e03d84b7787d33882cf72e1f15156dcfcea724a0cf0b02276a1966
Oct 31 01:37:27.447439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:27.447411    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:37:27.447523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:27.447474    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:37:27.447523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:27.447505    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:37:28.139421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:28.139389    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" event={"ID":"caf8aa23-f61d-475f-9752-80e896ec3eab","Type":"ContainerStarted","Data":"ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"}
Oct 31 01:37:28.139575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:28.139439    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" event={"ID":"caf8aa23-f61d-475f-9752-80e896ec3eab","Type":"ContainerStarted","Data":"2b5b0cc052e03d84b7787d33882cf72e1f15156dcfcea724a0cf0b02276a1966"}
Oct 31 01:37:28.356175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:28.356105    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podStartSLOduration=2.356089481 podStartE2EDuration="2.356089481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:28.305361294 +0000 UTC m=+1969.606317871" watchObservedRunningTime="2026-10-31 01:37:28.356089481 +0000 UTC m=+1969.657046058"
Oct 31 01:37:29.006593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:29.006563    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:29.012040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:29.012012    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:29.145644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:29.145618    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:29.146499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:29.146479    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:37:33.309512 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.309482    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"]
Oct 31 01:37:33.313720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.313701    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.416551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.416499    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"default-dockercfg-tx2g9\""
Oct 31 01:37:33.416551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.416526    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"kube-root-ca.crt\""
Oct 31 01:37:33.416551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.416533    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"tenantcfg-model-5e916d-kserve-self-signed-certs\""
Oct 31 01:37:33.416807 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.416562    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"openshift-service-ca.crt\""
Oct 31 01:37:33.421114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.421088    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"]
Oct 31 01:37:33.429801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.429775    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.429922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.429806    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.429965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.429924    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.430073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.430057    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcl5\" (UniqueName: \"kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.430108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.430094    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.430170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.430112    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531229    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531386    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcl5\" (UniqueName: \"kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531430    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531458    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531504    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531533    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531881    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.531924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531909    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.532128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.531933    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.533825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.533802    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.533945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.533928    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.588070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.587988    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcl5\" (UniqueName: \"kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5\") pod \"tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:33.627897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:33.627865    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:34.005179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.005132    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"]
Oct 31 01:37:34.007393 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:37:34.007365    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5288b28a_a422_49c2_aa82_8ad9879b7301.slice/crio-1ae537e6219b915d36fb5c1291e0a0ad60f9612e0eb74e31c3c79d4a21f5bcf0 WatchSource:0}: Error finding container 1ae537e6219b915d36fb5c1291e0a0ad60f9612e0eb74e31c3c79d4a21f5bcf0: Status 404 returned error can't find the container with id 1ae537e6219b915d36fb5c1291e0a0ad60f9612e0eb74e31c3c79d4a21f5bcf0
Oct 31 01:37:34.179993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.179955    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6" event={"ID":"5288b28a-a422-49c2-aa82-8ad9879b7301","Type":"ContainerStarted","Data":"5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e"}
Oct 31 01:37:34.179993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.179999    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6" event={"ID":"5288b28a-a422-49c2-aa82-8ad9879b7301","Type":"ContainerStarted","Data":"1ae537e6219b915d36fb5c1291e0a0ad60f9612e0eb74e31c3c79d4a21f5bcf0"}
Oct 31 01:37:34.297857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.297758    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6" podStartSLOduration=1.297744016 podStartE2EDuration="1.297744016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:34.294405538 +0000 UTC m=+1975.595362126" watchObservedRunningTime="2026-10-31 01:37:34.297744016 +0000 UTC m=+1975.598700593"
Oct 31 01:37:34.630212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.628226    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:34.744556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.744522    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"]
Oct 31 01:37:34.748617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.748595    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:34.802393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.802362    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-447154\""
Oct 31 01:37:34.802393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.802380    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-447154-dockercfg-fnb9n\""
Oct 31 01:37:34.806641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.806609    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"]
Oct 31 01:37:34.894628 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.884862    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zbn\" (UniqueName: \"kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn\") pod \"payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:34.894628 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.885043    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:34.986397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.986370    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:34.986585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.986493    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zbn\" (UniqueName: \"kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn\") pod \"payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:34.987095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.987072    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:34.989760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.989737    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"]
Oct 31 01:37:34.994636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.994611    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"]
Oct 31 01:37:34.994774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:34.994724    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.045010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.044974    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zbn\" (UniqueName: \"kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn\") pod \"payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:35.063330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.063304    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:35.087976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.087945    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.088128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.087991    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52xjh\" (UniqueName: \"kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh\") pod \"payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.188925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.188853    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.188925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.188888    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52xjh\" (UniqueName: \"kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh\") pod \"payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.189513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.189491    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.245351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.245316    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52xjh\" (UniqueName: \"kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh\") pod \"payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.312888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.312852    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:35.436818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.436792    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"]
Oct 31 01:37:35.438226 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:37:35.438201    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55384034_e4b0_43f0_a2a5_d6bf30c4a4ca.slice/crio-7e3905313424959af7373aa1bdf85cef2bc3630ebcd26e0721232fb1906de447 WatchSource:0}: Error finding container 7e3905313424959af7373aa1bdf85cef2bc3630ebcd26e0721232fb1906de447: Status 404 returned error can't find the container with id 7e3905313424959af7373aa1bdf85cef2bc3630ebcd26e0721232fb1906de447
Oct 31 01:37:35.689599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:35.689573    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"]
Oct 31 01:37:35.691919 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:37:35.691859    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1371773a_87d2_4be1_a0d0_4f5fc29e04ad.slice/crio-b82a4dcf47ecc068c6438e73572e7f81cc418e5ed7797e5c1de6c087ee77e87a WatchSource:0}: Error finding container b82a4dcf47ecc068c6438e73572e7f81cc418e5ed7797e5c1de6c087ee77e87a: Status 404 returned error can't find the container with id b82a4dcf47ecc068c6438e73572e7f81cc418e5ed7797e5c1de6c087ee77e87a
Oct 31 01:37:36.195029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.194998    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx" event={"ID":"1371773a-87d2-4be1-a0d0-4f5fc29e04ad","Type":"ContainerStarted","Data":"56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c"}
Oct 31 01:37:36.195250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.195046    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx" event={"ID":"1371773a-87d2-4be1-a0d0-4f5fc29e04ad","Type":"ContainerStarted","Data":"b82a4dcf47ecc068c6438e73572e7f81cc418e5ed7797e5c1de6c087ee77e87a"}
Oct 31 01:37:36.195250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.195105    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:36.196784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.196758    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff" event={"ID":"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca","Type":"ContainerStarted","Data":"47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c"}
Oct 31 01:37:36.196921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.196804    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff" event={"ID":"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca","Type":"ContainerStarted","Data":"7e3905313424959af7373aa1bdf85cef2bc3630ebcd26e0721232fb1906de447"}
Oct 31 01:37:36.196921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.196912    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:36.309078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.309028    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx" podStartSLOduration=2.309012043 podStartE2EDuration="2.309012043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:36.308195077 +0000 UTC m=+1977.609151654" watchObservedRunningTime="2026-10-31 01:37:36.309012043 +0000 UTC m=+1977.609968620"
Oct 31 01:37:36.431165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:36.425032    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff" podStartSLOduration=2.4250121780000002 podStartE2EDuration="2.425012178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:36.42456322 +0000 UTC m=+1977.725519801" watchObservedRunningTime="2026-10-31 01:37:36.425012178 +0000 UTC m=+1977.725968757"
Oct 31 01:37:40.193201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:40.193173    5419 scope.go:122] "RemoveContainer" containerID="544f4a8030848cc407fad98bb7fd66272c62a9e467d9c4a779f48c25760d399f"
Oct 31 01:37:43.628954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:43.628916    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:43.641564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:43.641539    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:44.271091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:44.271063    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:37:44.583567 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:37:44.583485    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:37:47.207986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:47.207956    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:37:47.208693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:47.208673    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:37:58.045838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:58.045798    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"]
Oct 31 01:37:58.094953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:58.094922    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da23bad0-fdmwp"]
Oct 31 01:37:58.837724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:37:58.837689    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9f33fe-54c3-4326-9621-7c0eddf586c5" path="/var/lib/kubelet/pods/1e9f33fe-54c3-4326-9621-7c0eddf586c5/volumes"
Oct 31 01:38:00.706509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.706471    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"]
Oct 31 01:38:00.710573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.710551    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:00.815009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.814974    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3d3023\""
Oct 31 01:38:00.815244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.815224    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3d3023-dockercfg-ctlnv\""
Oct 31 01:38:00.817679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.817661    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"]
Oct 31 01:38:00.834982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.834960    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bppz\" (UniqueName: \"kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:00.835104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.835005    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:00.936754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.936724    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:00.937018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.936987    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bppz\" (UniqueName: \"kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:00.937519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.937498    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:00.994208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:00.994181    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bppz\" (UniqueName: \"kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz\") pod \"payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:01.024355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.024334    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:01.403896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.403869    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"]
Oct 31 01:38:01.405491 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:38:01.405464    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291fb033_6e9a_458f_8d8c_7c4dc6293e51.slice/crio-316861e34c71c0f53427439859847c564e201b1f899bfd94e44c66f86922aa7c WatchSource:0}: Error finding container 316861e34c71c0f53427439859847c564e201b1f899bfd94e44c66f86922aa7c: Status 404 returned error can't find the container with id 316861e34c71c0f53427439859847c564e201b1f899bfd94e44c66f86922aa7c
Oct 31 01:38:01.495752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.495723    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"]
Oct 31 01:38:01.500081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.500061    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.557651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.557616    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"]
Oct 31 01:38:01.643786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.643754    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.643972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.643798    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b97b\" (UniqueName: \"kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b\") pod \"payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.745021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.744982    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.745450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.745045    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b97b\" (UniqueName: \"kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b\") pod \"payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.745710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.745679    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.803028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.802993    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b97b\" (UniqueName: \"kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b\") pod \"payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:01.813881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:01.813855    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:02.191943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.191918    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"]
Oct 31 01:38:02.194454 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:38:02.194423    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246ffb7e_71dc_4006_a855_d00d18dd0152.slice/crio-d18438e807a39d4bf18f4b199ee86bcf284bc4b8be404300d4a7f7d203b7d77f WatchSource:0}: Error finding container d18438e807a39d4bf18f4b199ee86bcf284bc4b8be404300d4a7f7d203b7d77f: Status 404 returned error can't find the container with id d18438e807a39d4bf18f4b199ee86bcf284bc4b8be404300d4a7f7d203b7d77f
Oct 31 01:38:02.382294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.382243    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz" event={"ID":"291fb033-6e9a-458f-8d8c-7c4dc6293e51","Type":"ContainerStarted","Data":"e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d"}
Oct 31 01:38:02.382489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.382344    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz" event={"ID":"291fb033-6e9a-458f-8d8c-7c4dc6293e51","Type":"ContainerStarted","Data":"316861e34c71c0f53427439859847c564e201b1f899bfd94e44c66f86922aa7c"}
Oct 31 01:38:02.382564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.382498    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:02.383900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.383864    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m" event={"ID":"246ffb7e-71dc-4006-a855-d00d18dd0152","Type":"ContainerStarted","Data":"f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87"}
Oct 31 01:38:02.384029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.383915    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m" event={"ID":"246ffb7e-71dc-4006-a855-d00d18dd0152","Type":"ContainerStarted","Data":"d18438e807a39d4bf18f4b199ee86bcf284bc4b8be404300d4a7f7d203b7d77f"}
Oct 31 01:38:02.384029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.383945    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:02.497074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.497022    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz" podStartSLOduration=2.497007289 podStartE2EDuration="2.497007289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:02.495881968 +0000 UTC m=+2003.796838546" watchObservedRunningTime="2026-10-31 01:38:02.497007289 +0000 UTC m=+2003.797963869"
Oct 31 01:38:02.663664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:02.647313    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m" podStartSLOduration=1.647288409 podStartE2EDuration="1.647288409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:02.630611958 +0000 UTC m=+2003.931568557" watchObservedRunningTime="2026-10-31 01:38:02.647288409 +0000 UTC m=+2003.948244987"
Oct 31 01:38:06.271261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:06.271222    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:38:13.393541 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:13.393507    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:38:13.393974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:13.393589    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:38:14.582155 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:38:14.582101    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:38:16.486357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:16.486311    5419 generic.go:356] "Generic (PLEG): container finished" podID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerID="e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca" exitCode=0
Oct 31 01:38:16.486898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:16.486454    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" event={"ID":"aa5b7129-09bc-45e9-b97a-5759d316f60f","Type":"ContainerDied","Data":"e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"}
Oct 31 01:38:16.487065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:16.487044    5419 scope.go:122] "RemoveContainer" containerID="e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"
Oct 31 01:38:16.491025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:16.490639    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:16.491025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:16.490973    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:16.491186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:16.491041    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:17.505978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:17.505943    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" event={"ID":"aa5b7129-09bc-45e9-b97a-5759d316f60f","Type":"ContainerStarted","Data":"7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280"}
Oct 31 01:38:17.610381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:17.610353    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:38:18.158553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:18.158515    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:38:18.159413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:18.159385    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:38:18.159548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:18.159468    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:38:19.158732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:19.158695    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:38:19.159266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:19.158750    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:38:20.158843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:20.158810    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:38:20.159266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:20.158871    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:38:21.158761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:21.158725    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:38:21.158979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:21.158786    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:38:22.136959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.136883    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"]
Oct 31 01:38:22.137183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.137126    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" containerID="cri-o://09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb" gracePeriod=30
Oct 31 01:38:22.158922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.158885    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 01:38:22.159070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.158946    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 01:38:22.201911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.201870    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"]
Oct 31 01:38:22.202178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.202128    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" containerID="cri-o://fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6" gracePeriod=30
Oct 31 01:38:22.205168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.205121    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"]
Oct 31 01:38:22.205325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.205306    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" containerID="cri-o://31b23d224f3083d2315881d354719713bbca50d22c5926897ff1010e93f05d35" gracePeriod=30
Oct 31 01:38:22.249073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.249038    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"]
Oct 31 01:38:22.249279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:22.249248    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" containerID="cri-o://7cb95175c97d691a272dfc8cff4f29969d4dd2da91f1a7aefbc9ee3b253a5643" gracePeriod=30
Oct 31 01:38:23.163750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.163720    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:38:23.164881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.164859    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:38:23.378829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.378798    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:38:23.378993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.378851    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:38:23.429408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.429327    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:38:23.429408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.429388    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:38:23.484642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.484605    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-5ks26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:38:23.484804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:23.484668    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:38:24.673815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:24.673780    5419 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75fd9f5d47-trv8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:38:24.674234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:24.673832    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:38:27.483564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.483536    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:38:27.563650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.563604    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:38:27.610029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.610001    5419 generic.go:356] "Generic (PLEG): container finished" podID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerID="7cb95175c97d691a272dfc8cff4f29969d4dd2da91f1a7aefbc9ee3b253a5643" exitCode=0
Oct 31 01:38:27.610192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.610074    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" event={"ID":"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61","Type":"ContainerDied","Data":"7cb95175c97d691a272dfc8cff4f29969d4dd2da91f1a7aefbc9ee3b253a5643"}
Oct 31 01:38:27.611560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.611538    5419 generic.go:356] "Generic (PLEG): container finished" podID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerID="fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6" exitCode=0
Oct 31 01:38:27.611687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.611596    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"
Oct 31 01:38:27.611687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.611624    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" event={"ID":"652d52af-d44a-4878-82a1-3e57b0be0b16","Type":"ContainerDied","Data":"fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6"}
Oct 31 01:38:27.611687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.611680    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z" event={"ID":"652d52af-d44a-4878-82a1-3e57b0be0b16","Type":"ContainerDied","Data":"7e904b10b885983b987f7049ac417f42ef1a52f6e345c70689889f48f9d29cfe"}
Oct 31 01:38:27.611868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.611711    5419 scope.go:122] "RemoveContainer" containerID="fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6"
Oct 31 01:38:27.612528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.612510    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:38:27.613195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.613173    5419 generic.go:356] "Generic (PLEG): container finished" podID="142e8bd0-8218-415c-accd-f6a82f66617d" containerID="31b23d224f3083d2315881d354719713bbca50d22c5926897ff1010e93f05d35" exitCode=0
Oct 31 01:38:27.613311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.613235    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" event={"ID":"142e8bd0-8218-415c-accd-f6a82f66617d","Type":"ContainerDied","Data":"31b23d224f3083d2315881d354719713bbca50d22c5926897ff1010e93f05d35"}
Oct 31 01:38:27.614667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.614649    5419 generic.go:356] "Generic (PLEG): container finished" podID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerID="09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb" exitCode=0
Oct 31 01:38:27.614770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.614685    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"
Oct 31 01:38:27.614770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.614707    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" event={"ID":"d0f52ca6-06ad-4fcd-b758-b530cf92c846","Type":"ContainerDied","Data":"09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb"}
Oct 31 01:38:27.614770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.614743    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26" event={"ID":"d0f52ca6-06ad-4fcd-b758-b530cf92c846","Type":"ContainerDied","Data":"1b3f04b4cc9afc5371cf90a0eaa00eb68da4519925e6e2dd693ebd542b594c06"}
Oct 31 01:38:27.616568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.616552    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:38:27.621657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.621641    5419 scope.go:122] "RemoveContainer" containerID="fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6"
Oct 31 01:38:27.621914 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:38:27.621898    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6\": container with ID starting with fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6 not found: ID does not exist" containerID="fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6"
Oct 31 01:38:27.621960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.621924    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6"} err="failed to get container status \"fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6\": rpc error: code = NotFound desc = could not find container \"fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6\": container with ID starting with fdaca84cad883e52fb976843fc925a901ebe838dff48ab15eea898f87802a0a6 not found: ID does not exist"
Oct 31 01:38:27.621960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.621940    5419 scope.go:122] "RemoveContainer" containerID="09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb"
Oct 31 01:38:27.630979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.630960    5419 scope.go:122] "RemoveContainer" containerID="09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb"
Oct 31 01:38:27.631270 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:38:27.631249    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb\": container with ID starting with 09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb not found: ID does not exist" containerID="09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb"
Oct 31 01:38:27.631327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.631277    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb"} err="failed to get container status \"09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb\": rpc error: code = NotFound desc = could not find container \"09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb\": container with ID starting with 09e07d1fe4e783b36177ab6d402ecceb551d738b398cd6fe842681a9962c5dfb not found: ID does not exist"
Oct 31 01:38:27.684993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.684958    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.685187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685008    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.685187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685051    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.685187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685083    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.685187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685118    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.685187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685183    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685211    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685244    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685284    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685315    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685306    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685347    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685352    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685407    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685427    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685450    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.685486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685485    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685523    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685570    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685580    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685622    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685657    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685701    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685775    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685788    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685718    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685813    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685848    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685894    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685926    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685960    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.685991    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686032    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.686062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686057    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.687000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686115    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.687000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686193    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.687000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686225    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.687000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686711    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.687000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.686863    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.687301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.687068    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.687371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.687296    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.689526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.688344    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.689526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.688950    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:27.689526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.689211    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.691104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.691055    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.691224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.691195    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.692049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.692008    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:38:27.692049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.692040    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:27.692244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.692099    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.692391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.692365    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "kube-api-access-79gnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.692463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.692421    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:38:27.692598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.692571    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.693089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.693062    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "kube-api-access-tpmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.693365 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.693195    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "kube-api-access-flhwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.694568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.694542    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.694870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.694847    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.694965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.694893    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:38:27.694965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.694945    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket\") pod \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\" (UID: \"d0f52ca6-06ad-4fcd-b758-b530cf92c846\") "
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.694979    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs\") pod \"652d52af-d44a-4878-82a1-3e57b0be0b16\" (UID: \"652d52af-d44a-4878-82a1-3e57b0be0b16\") "
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695005    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695033    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket\") pod \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\" (UID: \"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61\") "
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695069    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz\") pod \"142e8bd0-8218-415c-accd-f6a82f66617d\" (UID: \"142e8bd0-8218-415c-accd-f6a82f66617d\") "
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695097    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695239    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs" pod "652d52af-d44a-4878-82a1-3e57b0be0b16" (UID: "652d52af-d44a-4878-82a1-3e57b0be0b16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695267    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695350    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.695492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695395    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695519    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695847    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmrt\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-kube-api-access-tpmrt\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695864    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/652d52af-d44a-4878-82a1-3e57b0be0b16-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695875    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695875    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo" pod "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" (UID: "3b4ca327-19c4-4ab7-87f2-9fbf8179cd61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695883    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695928    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695944    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695961    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695978    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/142e8bd0-8218-415c-accd-f6a82f66617d-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.695993    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696007    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696020    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696034    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696048    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696061    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696075    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696089    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696103    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696117    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flhwh\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-kube-api-access-flhwh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696149    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696165    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696178    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696191    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696206    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79gnb\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-kube-api-access-79gnb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696221    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/142e8bd0-8218-415c-accd-f6a82f66617d-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696234    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696250    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/142e8bd0-8218-415c-accd-f6a82f66617d-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696265    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/652d52af-d44a-4878-82a1-3e57b0be0b16-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696279    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696293    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696307    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696321    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.696659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696334    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/652d52af-d44a-4878-82a1-3e57b0be0b16-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.697603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.696781    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:38:27.697603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.697122    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token" pod "d0f52ca6-06ad-4fcd-b758-b530cf92c846" (UID: "d0f52ca6-06ad-4fcd-b758-b530cf92c846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.697603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.697292    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz" pod "142e8bd0-8218-415c-accd-f6a82f66617d" (UID: "142e8bd0-8218-415c-accd-f6a82f66617d"). InnerVolumeSpecName "kube-api-access-frdrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:38:27.797675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.797642    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.797675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.797667    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdrz\" (UniqueName: \"kubernetes.io/projected/142e8bd0-8218-415c-accd-f6a82f66617d-kube-api-access-frdrz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.797675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.797677    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:27.797982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:27.797688    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f52ca6-06ad-4fcd-b758-b530cf92c846-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:38:28.171247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.171202    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"]
Oct 31 01:38:28.181205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.181179    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-trv8z"]
Oct 31 01:38:28.187506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.187484    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"]
Oct 31 01:38:28.188075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188057    5419 cpu_manager.go:414] "containerMap: removing container" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy"
Oct 31 01:38:28.188159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188078    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy"
Oct 31 01:38:28.188159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188089    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy"
Oct 31 01:38:28.188159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188094    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy"
Oct 31 01:38:28.188159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188113    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy"
Oct 31 01:38:28.188159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188120    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy"
Oct 31 01:38:28.188159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188156    5419 cpu_manager.go:414] "containerMap: removing container" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy"
Oct 31 01:38:28.188339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188162    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy"
Oct 31 01:38:28.188339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188244    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" containerName="istio-proxy"
Oct 31 01:38:28.188339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188255    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" containerName="istio-proxy"
Oct 31 01:38:28.188339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188261    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" containerName="istio-proxy"
Oct 31 01:38:28.188339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.188270    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" containerName="istio-proxy"
Oct 31 01:38:28.193015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.193000    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.288515 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.288480    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-df5d0b-openshift-default-dockercfg-xs4v4\""
Oct 31 01:38:28.317677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.315191    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"]
Oct 31 01:38:28.322555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322506    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322623    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw7j2\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322657    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322718    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322750    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322790    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322820    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.322870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322863    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.323421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.322912    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.324981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.324953    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"]
Oct 31 01:38:28.347624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.347586    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5ks26"]
Oct 31 01:38:28.434242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434107    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7j2\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434229    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434309    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434385    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434446    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434479    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434524    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434576    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.434741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.434685    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.469844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.435188    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.469844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.435834    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.469844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.436094    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.469844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.436938    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.469844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.449517    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.472869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.472703    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.474490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.474467    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.494733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.494700    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7j2\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.498971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.498939    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token\") pod \"e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") " pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.505905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.505880    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:28.622548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.622514    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"
Oct 31 01:38:28.622548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.622543    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb" event={"ID":"3b4ca327-19c4-4ab7-87f2-9fbf8179cd61","Type":"ContainerDied","Data":"5d6a02d7542ac3fdf3f1742ca27ab0cfd26648aa31d4dd804121a431c7b482ee"}
Oct 31 01:38:28.622783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.622604    5419 scope.go:122] "RemoveContainer" containerID="7cb95175c97d691a272dfc8cff4f29969d4dd2da91f1a7aefbc9ee3b253a5643"
Oct 31 01:38:28.625964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.625644    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt" event={"ID":"142e8bd0-8218-415c-accd-f6a82f66617d","Type":"ContainerDied","Data":"58aea93ef0429103468d87f5a35d5829b8ec05241699c81c896b0495b515f326"}
Oct 31 01:38:28.625964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.625667    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"
Oct 31 01:38:28.636234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.636132    5419 scope.go:122] "RemoveContainer" containerID="31b23d224f3083d2315881d354719713bbca50d22c5926897ff1010e93f05d35"
Oct 31 01:38:28.786986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.786955    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"]
Oct 31 01:38:28.792646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.792615    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-jg8xt"]
Oct 31 01:38:28.838447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.838418    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142e8bd0-8218-415c-accd-f6a82f66617d" path="/var/lib/kubelet/pods/142e8bd0-8218-415c-accd-f6a82f66617d/volumes"
Oct 31 01:38:28.838928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.838898    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652d52af-d44a-4878-82a1-3e57b0be0b16" path="/var/lib/kubelet/pods/652d52af-d44a-4878-82a1-3e57b0be0b16/volumes"
Oct 31 01:38:28.839382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.839367    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f52ca6-06ad-4fcd-b758-b530cf92c846" path="/var/lib/kubelet/pods/d0f52ca6-06ad-4fcd-b758-b530cf92c846/volumes"
Oct 31 01:38:28.899844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.899806    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"]
Oct 31 01:38:28.903794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.903769    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75fd9f5d47-5tdsb"]
Oct 31 01:38:28.946350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.946324    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"]
Oct 31 01:38:28.948382 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:38:28.948360    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9aa2ebd_853c_45dd_a273_b0aae3d34984.slice/crio-4b313e07ba9c4004018695b24897adb7ba9d06c1e98641a2c5526347bc801c4d WatchSource:0}: Error finding container 4b313e07ba9c4004018695b24897adb7ba9d06c1e98641a2c5526347bc801c4d: Status 404 returned error can't find the container with id 4b313e07ba9c4004018695b24897adb7ba9d06c1e98641a2c5526347bc801c4d
Oct 31 01:38:28.950876 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.950847    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:28.950955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.950911    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:28.950955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:28.950938    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:29.644528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:29.644486    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" event={"ID":"e9aa2ebd-853c-45dd-a273-b0aae3d34984","Type":"ContainerStarted","Data":"467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"}
Oct 31 01:38:29.645126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:29.644559    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" event={"ID":"e9aa2ebd-853c-45dd-a273-b0aae3d34984","Type":"ContainerStarted","Data":"4b313e07ba9c4004018695b24897adb7ba9d06c1e98641a2c5526347bc801c4d"}
Oct 31 01:38:29.763156 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:29.763079    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podStartSLOduration=1.763065846 podStartE2EDuration="1.763065846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:29.760818236 +0000 UTC m=+2031.061774816" watchObservedRunningTime="2026-10-31 01:38:29.763065846 +0000 UTC m=+2031.064022423"
Oct 31 01:38:30.506390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:30.506353    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:30.512030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:30.512005    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:30.653063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:30.653030    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:30.653922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:30.653905    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:38:30.839241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:30.839167    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4ca327-19c4-4ab7-87f2-9fbf8179cd61" path="/var/lib/kubelet/pods/3b4ca327-19c4-4ab7-87f2-9fbf8179cd61/volumes"
Oct 31 01:38:40.237949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:40.237915    5419 scope.go:122] "RemoveContainer" containerID="710f8fcc14f550e8703e6902c20a1a19a9fef83bfc4a4cdcd17dffd9b2543df5"
Oct 31 01:38:40.248709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:40.248691    5419 scope.go:122] "RemoveContainer" containerID="b2ef38b9681964f24af375e7300875b945f891c6a231c09eaef7a47d1e952f9e"
Oct 31 01:38:44.582765 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:38:44.582722    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:38:48.465699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.465664    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"]
Oct 31 01:38:48.471540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.471516    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.506451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.506419    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckgh\" (UniqueName: \"kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.506598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.506556    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.527388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.527363    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"]
Oct 31 01:38:48.603164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.582557    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-df5d0b-dockercfg-dwdnw\""
Oct 31 01:38:48.606203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.587871    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-df5d0b\""
Oct 31 01:38:48.611160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.607652    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.611160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.607805    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckgh\" (UniqueName: \"kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.611160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.609207    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.637878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.637830    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"]
Oct 31 01:38:48.643469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.643442    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.664193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.664164    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckgh\" (UniqueName: \"kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh\") pod \"payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.686347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.686319    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"]
Oct 31 01:38:48.708869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.708815    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.709104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.709079    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxxv\" (UniqueName: \"kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv\") pod \"payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.784922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.784876    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:48.810477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.810445    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxxv\" (UniqueName: \"kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv\") pod \"payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.810625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.810555    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.811231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.811211    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.868175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.866678    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxxv\" (UniqueName: \"kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv\") pod \"payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:48.962783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:48.962752    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:49.163937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.163910    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"]
Oct 31 01:38:49.165305 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:38:49.165276    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf12a6f3_5ccc_429b_bb15_3660e65d091a.slice/crio-e52a064d164924a3e63415ecaf62fc5e05793ba7a9e751288218a460ffdc8bd8 WatchSource:0}: Error finding container e52a064d164924a3e63415ecaf62fc5e05793ba7a9e751288218a460ffdc8bd8: Status 404 returned error can't find the container with id e52a064d164924a3e63415ecaf62fc5e05793ba7a9e751288218a460ffdc8bd8
Oct 31 01:38:49.293099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.293057    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"]
Oct 31 01:38:49.297212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.297191    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.322931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.322893    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.322944    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323024    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323100    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323214 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323199    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323241    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323280    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323326    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.323395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.323386    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cpg\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.344572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.344528    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"]
Oct 31 01:38:49.346649 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:38:49.346599    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60630003_e9bf_416e_bc71_ad136e79c8cc.slice/crio-c32ff29458d9017cc4fa165e656b31dcaff99117f57d977455060563c8baf634 WatchSource:0}: Error finding container c32ff29458d9017cc4fa165e656b31dcaff99117f57d977455060563c8baf634: Status 404 returned error can't find the container with id c32ff29458d9017cc4fa165e656b31dcaff99117f57d977455060563c8baf634
Oct 31 01:38:49.370942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.370910    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8af859-openshift-default-dockercfg-2ft7r\""
Oct 31 01:38:49.394722 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.394687    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"]
Oct 31 01:38:49.424780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.424745    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.424914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.424828    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.424914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.424884    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.424928    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.424968    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425015    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425066    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cpg\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425122    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425174    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425283    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425293    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425552    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425723    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.425943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.425917    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.428195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.428172    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.428497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.428479    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.501239 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.485447    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.533969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.533924    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cpg\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.615192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.615108    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:49.619838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.619805    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:38:49.826309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.825195    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m" event={"ID":"60630003-e9bf-416e-bc71-ad136e79c8cc","Type":"ContainerStarted","Data":"0c8478c998d34067f394e30a63f54e5903eb4f788684dac94977b095493d4aef"}
Oct 31 01:38:49.826309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.825259    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m" event={"ID":"60630003-e9bf-416e-bc71-ad136e79c8cc","Type":"ContainerStarted","Data":"c32ff29458d9017cc4fa165e656b31dcaff99117f57d977455060563c8baf634"}
Oct 31 01:38:49.826309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.826225    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:38:49.841997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.840842    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv" event={"ID":"cf12a6f3-5ccc-429b-bb15-3660e65d091a","Type":"ContainerStarted","Data":"99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1"}
Oct 31 01:38:49.841997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.840912    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv" event={"ID":"cf12a6f3-5ccc-429b-bb15-3660e65d091a","Type":"ContainerStarted","Data":"e52a064d164924a3e63415ecaf62fc5e05793ba7a9e751288218a460ffdc8bd8"}
Oct 31 01:38:49.841997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.841706    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:38:49.942854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:49.942752    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m" podStartSLOduration=1.942736105 podStartE2EDuration="1.942736105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:49.940845352 +0000 UTC m=+2051.241801930" watchObservedRunningTime="2026-10-31 01:38:49.942736105 +0000 UTC m=+2051.243692682"
Oct 31 01:38:50.080874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.080797    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv" podStartSLOduration=2.080774873 podStartE2EDuration="2.080774873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:50.072847805 +0000 UTC m=+2051.373804384" watchObservedRunningTime="2026-10-31 01:38:50.080774873 +0000 UTC m=+2051.381731451"
Oct 31 01:38:50.103292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.093320    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"]
Oct 31 01:38:50.103464 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:38:50.103426    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ddfed5_fee3_49a2_8ce9_baef7e9882bc.slice/crio-6bd0a8620b11c95b39c3288d4d1a4a3e577c92560fd3a904f44d8ce405556303 WatchSource:0}: Error finding container 6bd0a8620b11c95b39c3288d4d1a4a3e577c92560fd3a904f44d8ce405556303: Status 404 returned error can't find the container with id 6bd0a8620b11c95b39c3288d4d1a4a3e577c92560fd3a904f44d8ce405556303
Oct 31 01:38:50.106311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.106278    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:50.106400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.106347    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:50.106400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.106377    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:50.848507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.848410    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" event={"ID":"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc","Type":"ContainerStarted","Data":"c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"}
Oct 31 01:38:50.848507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.848464    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" event={"ID":"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc","Type":"ContainerStarted","Data":"6bd0a8620b11c95b39c3288d4d1a4a3e577c92560fd3a904f44d8ce405556303"}
Oct 31 01:38:50.968708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:50.968647    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podStartSLOduration=1.968625722 podStartE2EDuration="1.968625722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:50.96758867 +0000 UTC m=+2052.268545265" watchObservedRunningTime="2026-10-31 01:38:50.968625722 +0000 UTC m=+2052.269582300"
Oct 31 01:38:51.616603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:51.616567    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:51.621765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:51.621739    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:51.854885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:51.854856    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:51.855865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:51.855847    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:38:59.182684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.181029    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": read tcp 10.133.0.2:41108->10.133.0.168:15021: read: connection reset by peer" start-of-body=
Oct 31 01:38:59.182684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.181087    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": read tcp 10.133.0.2:41108->10.133.0.168:15021: read: connection reset by peer"
Oct 31 01:38:59.933720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.933690    5419 generic.go:356] "Generic (PLEG): container finished" podID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerID="ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee" exitCode=0
Oct 31 01:38:59.933989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.933758    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" event={"ID":"caf8aa23-f61d-475f-9752-80e896ec3eab","Type":"ContainerDied","Data":"ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"}
Oct 31 01:38:59.934162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.934148    5419 scope.go:122] "RemoveContainer" containerID="ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"
Oct 31 01:38:59.935600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.935565    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:59.935672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.935643    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:38:59.935719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:38:59.935674    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:39:00.942031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:00.941999    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" event={"ID":"caf8aa23-f61d-475f-9752-80e896ec3eab","Type":"ContainerStarted","Data":"63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e"}
Oct 31 01:39:01.006776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:01.006751    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:39:01.006987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:01.006960    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:39:01.007058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:01.007029    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 01:39:01.858818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:01.858784    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:39:01.860233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:01.860209    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:39:02.015169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.013236    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:39:02.015169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.013291    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 01:39:02.220338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.220304    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"]
Oct 31 01:39:02.233422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.233387    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.286455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.286424    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8af859\""
Oct 31 01:39:02.287546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.287519    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8af859-dockercfg-l5sld\""
Oct 31 01:39:02.295799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.295766    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"]
Oct 31 01:39:02.405210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.398810    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.413446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.413413    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbkw\" (UniqueName: \"kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw\") pod \"payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.551174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.551070    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbkw\" (UniqueName: \"kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw\") pod \"payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.551333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.551214    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.551921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.551901    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.607714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.607680    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbkw\" (UniqueName: \"kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw\") pod \"payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:02.857860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:02.857782    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:03.006978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:03.006939    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:39:03.008111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:03.008069    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 01:39:03.267235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:03.266952    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"]
Oct 31 01:39:03.269348 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:39:03.269314    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4d041f_2f27_4b12_bfc5_bb7b3bea2b76.slice/crio-ab25f095bd64a602a7059b360de2b2bdea312215122cc3288a8f83cbc6fe2e2d WatchSource:0}: Error finding container ab25f095bd64a602a7059b360de2b2bdea312215122cc3288a8f83cbc6fe2e2d: Status 404 returned error can't find the container with id ab25f095bd64a602a7059b360de2b2bdea312215122cc3288a8f83cbc6fe2e2d
Oct 31 01:39:04.006342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:04.006303    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8" event={"ID":"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76","Type":"ContainerStarted","Data":"8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616"}
Oct 31 01:39:04.006562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:04.006366    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8" event={"ID":"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76","Type":"ContainerStarted","Data":"ab25f095bd64a602a7059b360de2b2bdea312215122cc3288a8f83cbc6fe2e2d"}
Oct 31 01:39:04.006562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:04.006395    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:04.006748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:04.006719    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:39:04.006818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:04.006793    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 01:39:04.122164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:04.122101    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8" podStartSLOduration=2.1220871 podStartE2EDuration="2.1220871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:04.119252804 +0000 UTC m=+2065.420209382" watchObservedRunningTime="2026-10-31 01:39:04.1220871 +0000 UTC m=+2065.423043696"
Oct 31 01:39:05.006781 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:05.006735    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:39:05.007237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:05.006799    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 01:39:06.007116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:06.007080    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 01:39:06.007552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:06.007149    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 01:39:07.010735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:07.010703    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:39:07.011172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:07.011004    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:39:07.011728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:07.011709    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:39:14.582928 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:39:14.582881    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:39:14.848683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:14.848602    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"]
Oct 31 01:39:14.962807 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:14.962760    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"]
Oct 31 01:39:15.016614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:15.016585    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:39:19.849936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:19.849843    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk" podUID="1e37029e-a52f-49be-9447-80d93268c132" containerName="payload-processing" containerID="cri-o://cabeb97ff65fb4df92094304d032060d725e1ae3c433f2bc7b87b61d94b651b3" gracePeriod=25
Oct 31 01:39:19.963709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:19.963666    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz" podUID="29194385-841c-4ed1-b17f-f765408fd255" containerName="payload-pre-processing" containerID="cri-o://f5cbced8ec0ebdb4b88b822f0e7dd863d1ea986aac1e62983b7feafe8e23a014" gracePeriod=25
Oct 31 01:39:20.121995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.121962    5419 generic.go:356] "Generic (PLEG): container finished" podID="1e37029e-a52f-49be-9447-80d93268c132" containerID="cabeb97ff65fb4df92094304d032060d725e1ae3c433f2bc7b87b61d94b651b3" exitCode=0
Oct 31 01:39:20.122163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.122033    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk" event={"ID":"1e37029e-a52f-49be-9447-80d93268c132","Type":"ContainerDied","Data":"cabeb97ff65fb4df92094304d032060d725e1ae3c433f2bc7b87b61d94b651b3"}
Oct 31 01:39:20.122163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.122084    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk" event={"ID":"1e37029e-a52f-49be-9447-80d93268c132","Type":"ContainerDied","Data":"c2dd7ae5ca141d08d8d3c229bdd69a72e6ca893be91a85005a1f56cad72c043e"}
Oct 31 01:39:20.122163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.122105    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2dd7ae5ca141d08d8d3c229bdd69a72e6ca893be91a85005a1f56cad72c043e"
Oct 31 01:39:20.123665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.123638    5419 generic.go:356] "Generic (PLEG): container finished" podID="29194385-841c-4ed1-b17f-f765408fd255" containerID="f5cbced8ec0ebdb4b88b822f0e7dd863d1ea986aac1e62983b7feafe8e23a014" exitCode=0
Oct 31 01:39:20.123773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.123710    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz" event={"ID":"29194385-841c-4ed1-b17f-f765408fd255","Type":"ContainerDied","Data":"f5cbced8ec0ebdb4b88b822f0e7dd863d1ea986aac1e62983b7feafe8e23a014"}
Oct 31 01:39:20.138472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.138451    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:39:20.194347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.194319    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume\") pod \"1e37029e-a52f-49be-9447-80d93268c132\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") "
Oct 31 01:39:20.194347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.194360    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp\" (UniqueName: \"kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp\") pod \"1e37029e-a52f-49be-9447-80d93268c132\" (UID: \"1e37029e-a52f-49be-9447-80d93268c132\") "
Oct 31 01:39:20.194820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.194791    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume" pod "1e37029e-a52f-49be-9447-80d93268c132" (UID: "1e37029e-a52f-49be-9447-80d93268c132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:39:20.196842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.196818    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp" pod "1e37029e-a52f-49be-9447-80d93268c132" (UID: "1e37029e-a52f-49be-9447-80d93268c132"). InnerVolumeSpecName "kube-api-access-pdpkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:39:20.214800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.214774    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:39:20.295880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.295835    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf\" (UniqueName: \"kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf\") pod \"29194385-841c-4ed1-b17f-f765408fd255\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") "
Oct 31 01:39:20.296056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.296003    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume\") pod \"29194385-841c-4ed1-b17f-f765408fd255\" (UID: \"29194385-841c-4ed1-b17f-f765408fd255\") "
Oct 31 01:39:20.296442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.296418    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume" pod "29194385-841c-4ed1-b17f-f765408fd255" (UID: "29194385-841c-4ed1-b17f-f765408fd255"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:39:20.296551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.296538    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e37029e-a52f-49be-9447-80d93268c132-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:20.296587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.296554    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdpkp\" (UniqueName: \"kubernetes.io/projected/1e37029e-a52f-49be-9447-80d93268c132-kube-api-access-pdpkp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:20.296587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.296563    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29194385-841c-4ed1-b17f-f765408fd255-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:20.298042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.298022    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf" pod "29194385-841c-4ed1-b17f-f765408fd255" (UID: "29194385-841c-4ed1-b17f-f765408fd255"). InnerVolumeSpecName "kube-api-access-mk2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:39:20.397575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:20.397487    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2wf\" (UniqueName: \"kubernetes.io/projected/29194385-841c-4ed1-b17f-f765408fd255-kube-api-access-mk2wf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:39:21.131549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.131459    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"
Oct 31 01:39:21.131549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.131505    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz" event={"ID":"29194385-841c-4ed1-b17f-f765408fd255","Type":"ContainerDied","Data":"92ba322a98b9688e17b88fbd97cfa7a9834ad41c40fc64382951e40c590bcd86"}
Oct 31 01:39:21.132121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.131569    5419 scope.go:122] "RemoveContainer" containerID="f5cbced8ec0ebdb4b88b822f0e7dd863d1ea986aac1e62983b7feafe8e23a014"
Oct 31 01:39:21.132121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.131730    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"
Oct 31 01:39:21.293762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.293731    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"]
Oct 31 01:39:21.343504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.343468    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-59644f7dc9-6bbsz"]
Oct 31 01:39:21.450820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.450733    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"]
Oct 31 01:39:21.455687 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:21.455656    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-6f5cbc65ff-k2nmk"]
Oct 31 01:39:22.837081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:22.837051    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e37029e-a52f-49be-9447-80d93268c132" path="/var/lib/kubelet/pods/1e37029e-a52f-49be-9447-80d93268c132/volumes"
Oct 31 01:39:22.837518 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:22.837503    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29194385-841c-4ed1-b17f-f765408fd255" path="/var/lib/kubelet/pods/29194385-841c-4ed1-b17f-f765408fd255/volumes"
Oct 31 01:39:34.941341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:34.941306    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:39:40.294302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:40.294273    5419 scope.go:122] "RemoveContainer" containerID="f273156dc8e53a6b4c15715def32be7cd0ffd4b171c8e3456028d1d677ad3262"
Oct 31 01:39:40.303948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:40.303927    5419 scope.go:122] "RemoveContainer" containerID="cc4143bc92aac7111a0a2f34ef530137a69327f8f654bab379f8ac78052193cc"
Oct 31 01:39:40.312758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:40.312737    5419 scope.go:122] "RemoveContainer" containerID="8db1e08c900f60e2fee63489047901b36206d1a7f9bb615771d9f7a963cd9c94"
Oct 31 01:39:40.321090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:40.321067    5419 scope.go:122] "RemoveContainer" containerID="bdd79342e74701c7008d61b4d3da1d80ada7d4e72f210f289d57d55ccc990523"
Oct 31 01:39:40.329303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:40.329286    5419 scope.go:122] "RemoveContainer" containerID="04f718dacfe8f70712b7328a83c3c0b7eef16eb1a56e2fbff784385c3966f059"
Oct 31 01:39:44.648765 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:39:44.648720    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:39:50.821948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.821906    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"]
Oct 31 01:39:50.822732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.822710    5419 cpu_manager.go:414] "containerMap: removing container" podUID="29194385-841c-4ed1-b17f-f765408fd255" containerName="payload-pre-processing"
Oct 31 01:39:50.822843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.822734    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29194385-841c-4ed1-b17f-f765408fd255" containerName="payload-pre-processing"
Oct 31 01:39:50.822843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.822780    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1e37029e-a52f-49be-9447-80d93268c132" containerName="payload-processing"
Oct 31 01:39:50.822843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.822788    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e37029e-a52f-49be-9447-80d93268c132" containerName="payload-processing"
Oct 31 01:39:50.823017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.822972    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="29194385-841c-4ed1-b17f-f765408fd255" containerName="payload-pre-processing"
Oct 31 01:39:50.823017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.822995    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e37029e-a52f-49be-9447-80d93268c132" containerName="payload-processing"
Oct 31 01:39:50.828192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.828172    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:50.881277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.881248    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwq2\" (UniqueName: \"kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:50.881451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.881406    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:50.884702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.884674    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"]
Oct 31 01:39:50.890503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.890470    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-99c8bd\""
Oct 31 01:39:50.929158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.929118    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-99c8bd-dockercfg-ltxtg\""
Oct 31 01:39:50.983036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.983008    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:50.983206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.983160    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwq2\" (UniqueName: \"kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:50.983716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:50.983696    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:51.040649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.040619    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwq2\" (UniqueName: \"kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2\") pod \"payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:51.141848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.141763    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:51.516281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.516249    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"]
Oct 31 01:39:51.517848 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:39:51.517822    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc34e482_597e_4030_8d0f_bcd06daceede.slice/crio-0442f5d8608764d22fda7bbe35be0c8877991521e8cee08bb57512c074773604 WatchSource:0}: Error finding container 0442f5d8608764d22fda7bbe35be0c8877991521e8cee08bb57512c074773604: Status 404 returned error can't find the container with id 0442f5d8608764d22fda7bbe35be0c8877991521e8cee08bb57512c074773604
Oct 31 01:39:51.605868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.605846    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"]
Oct 31 01:39:51.609853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.609837    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.666612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.666584    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"]
Oct 31 01:39:51.691522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.691496    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9rq\" (UniqueName: \"kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq\") pod \"payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.691694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.691658    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.793445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.793372    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9rq\" (UniqueName: \"kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq\") pod \"payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.793582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.793497    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.794127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.794108    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.850111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.850085    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9rq\" (UniqueName: \"kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq\") pod \"payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:51.923987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:51.923956    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:52.298173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:52.298129    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"]
Oct 31 01:39:52.300264 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:39:52.300237    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a4f2d4_9a54_471f_8f4e_16bee6db1d09.slice/crio-202afd4e5f672c24fc04878d90adade9a6cf76b22e33ac714543e905226c8718 WatchSource:0}: Error finding container 202afd4e5f672c24fc04878d90adade9a6cf76b22e33ac714543e905226c8718: Status 404 returned error can't find the container with id 202afd4e5f672c24fc04878d90adade9a6cf76b22e33ac714543e905226c8718
Oct 31 01:39:52.349083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:52.349044    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t" event={"ID":"cc34e482-597e-4030-8d0f-bcd06daceede","Type":"ContainerStarted","Data":"36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb"}
Oct 31 01:39:52.349263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:52.349107    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t" event={"ID":"cc34e482-597e-4030-8d0f-bcd06daceede","Type":"ContainerStarted","Data":"0442f5d8608764d22fda7bbe35be0c8877991521e8cee08bb57512c074773604"}
Oct 31 01:39:52.349263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:52.349188    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:39:52.350653 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:52.350583    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" event={"ID":"38a4f2d4-9a54-471f-8f4e-16bee6db1d09","Type":"ContainerStarted","Data":"202afd4e5f672c24fc04878d90adade9a6cf76b22e33ac714543e905226c8718"}
Oct 31 01:39:52.463094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:52.463044    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t" podStartSLOduration=2.463025549 podStartE2EDuration="2.463025549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:52.462091404 +0000 UTC m=+2113.763047980" watchObservedRunningTime="2026-10-31 01:39:52.463025549 +0000 UTC m=+2113.763982121"
Oct 31 01:39:53.359527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:53.359494    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" event={"ID":"38a4f2d4-9a54-471f-8f4e-16bee6db1d09","Type":"ContainerStarted","Data":"14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"}
Oct 31 01:39:53.359982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:53.359774    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:39:53.477801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:53.477747    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" podStartSLOduration=2.477732849 podStartE2EDuration="2.477732849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:53.473597374 +0000 UTC m=+2114.774553962" watchObservedRunningTime="2026-10-31 01:39:53.477732849 +0000 UTC m=+2114.778689425"
Oct 31 01:39:59.871173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:59.871111    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"]
Oct 31 01:39:59.958866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:39:59.958827    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"]
Oct 31 01:40:02.434282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:02.434255    5419 generic.go:356] "Generic (PLEG): container finished" podID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerID="14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238" exitCode=1
Oct 31 01:40:02.434575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:02.434329    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" event={"ID":"38a4f2d4-9a54-471f-8f4e-16bee6db1d09","Type":"ContainerDied","Data":"14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"}
Oct 31 01:40:02.434768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:02.434754    5419 scope.go:122] "RemoveContainer" containerID="14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"
Oct 31 01:40:03.361375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:03.361348    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:40:03.442678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:03.442645    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" event={"ID":"38a4f2d4-9a54-471f-8f4e-16bee6db1d09","Type":"ContainerStarted","Data":"1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689"}
Oct 31 01:40:03.443201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:03.442888    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:40:04.872419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:04.872384    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx" podUID="1371773a-87d2-4be1-a0d0-4f5fc29e04ad" containerName="payload-processing" containerID="cri-o://56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c" gracePeriod=25
Oct 31 01:40:04.960104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:04.960072    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff" podUID="55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" containerName="payload-pre-processing" containerID="cri-o://47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c" gracePeriod=25
Oct 31 01:40:05.225265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.225239    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:40:05.259400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.259378    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:40:05.267093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267070    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume\") pod \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") "
Oct 31 01:40:05.267306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267290    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume\") pod \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") "
Oct 31 01:40:05.267357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267347    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh\" (UniqueName: \"kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh\") pod \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\" (UID: \"1371773a-87d2-4be1-a0d0-4f5fc29e04ad\") "
Oct 31 01:40:05.267410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267388    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume" pod "55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" (UID: "55384034-e4b0-43f0-a2a5-d6bf30c4a4ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:05.267458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267393    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn\" (UniqueName: \"kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn\") pod \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\" (UID: \"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca\") "
Oct 31 01:40:05.267655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267635    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume" pod "1371773a-87d2-4be1-a0d0-4f5fc29e04ad" (UID: "1371773a-87d2-4be1-a0d0-4f5fc29e04ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:40:05.267995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267973    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:05.267995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.267995    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:05.269639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.269621    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh" pod "1371773a-87d2-4be1-a0d0-4f5fc29e04ad" (UID: "1371773a-87d2-4be1-a0d0-4f5fc29e04ad"). InnerVolumeSpecName "kube-api-access-52xjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:05.269721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.269682    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn" pod "55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" (UID: "55384034-e4b0-43f0-a2a5-d6bf30c4a4ca"). InnerVolumeSpecName "kube-api-access-27zbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:40:05.369485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.369451    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52xjh\" (UniqueName: \"kubernetes.io/projected/1371773a-87d2-4be1-a0d0-4f5fc29e04ad-kube-api-access-52xjh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:05.369485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.369478    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27zbn\" (UniqueName: \"kubernetes.io/projected/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca-kube-api-access-27zbn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:40:05.484655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.484618    5419 generic.go:356] "Generic (PLEG): container finished" podID="1371773a-87d2-4be1-a0d0-4f5fc29e04ad" containerID="56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c" exitCode=0
Oct 31 01:40:05.484852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.484666    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"
Oct 31 01:40:05.484852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.484700    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx" event={"ID":"1371773a-87d2-4be1-a0d0-4f5fc29e04ad","Type":"ContainerDied","Data":"56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c"}
Oct 31 01:40:05.484852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.484764    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx" event={"ID":"1371773a-87d2-4be1-a0d0-4f5fc29e04ad","Type":"ContainerDied","Data":"b82a4dcf47ecc068c6438e73572e7f81cc418e5ed7797e5c1de6c087ee77e87a"}
Oct 31 01:40:05.484852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.484791    5419 scope.go:122] "RemoveContainer" containerID="56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c"
Oct 31 01:40:05.489637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.489613    5419 generic.go:356] "Generic (PLEG): container finished" podID="55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" containerID="47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c" exitCode=0
Oct 31 01:40:05.489766 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.489734    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff" event={"ID":"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca","Type":"ContainerDied","Data":"47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c"}
Oct 31 01:40:05.489836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.489813    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff" event={"ID":"55384034-e4b0-43f0-a2a5-d6bf30c4a4ca","Type":"ContainerDied","Data":"7e3905313424959af7373aa1bdf85cef2bc3630ebcd26e0721232fb1906de447"}
Oct 31 01:40:05.489930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.489915    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"
Oct 31 01:40:05.499274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.499203    5419 scope.go:122] "RemoveContainer" containerID="56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c"
Oct 31 01:40:05.501677 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:40:05.501641    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c\": container with ID starting with 56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c not found: ID does not exist" containerID="56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c"
Oct 31 01:40:05.502315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.501679    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c"} err="failed to get container status \"56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c\": rpc error: code = NotFound desc = could not find container \"56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c\": container with ID starting with 56b03839f88c0dfe13d1ebdb9e510ae313e7237af19102c5a8979e2f4680e14c not found: ID does not exist"
Oct 31 01:40:05.502315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.501704    5419 scope.go:122] "RemoveContainer" containerID="47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c"
Oct 31 01:40:05.513544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.513524    5419 scope.go:122] "RemoveContainer" containerID="47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c"
Oct 31 01:40:05.513885 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:40:05.513866    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c\": container with ID starting with 47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c not found: ID does not exist" containerID="47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c"
Oct 31 01:40:05.513945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.513897    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c"} err="failed to get container status \"47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c\": rpc error: code = NotFound desc = could not find container \"47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c\": container with ID starting with 47d8904c3ca4330fac7d3869a78efaea33a9fe9c291d50ea35598ff603fe849c not found: ID does not exist"
Oct 31 01:40:05.653748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.653710    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"]
Oct 31 01:40:05.656296 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.656267    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-447154-56895c4795-7g5ff"]
Oct 31 01:40:05.762430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.762334    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"]
Oct 31 01:40:05.764731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:05.764711    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-447154-78d7bc47bf-gfzrx"]
Oct 31 01:40:06.837959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:06.837926    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1371773a-87d2-4be1-a0d0-4f5fc29e04ad" path="/var/lib/kubelet/pods/1371773a-87d2-4be1-a0d0-4f5fc29e04ad/volumes"
Oct 31 01:40:06.838402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:06.838355    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" path="/var/lib/kubelet/pods/55384034-e4b0-43f0-a2a5-d6bf30c4a4ca/volumes"
Oct 31 01:40:11.613504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:11.613468    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:40:14.452222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:14.452190    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:40:14.608960 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:40:14.608916    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:40:22.469891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.469837    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"]
Oct 31 01:40:22.470694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.470674    5419 cpu_manager.go:414] "containerMap: removing container" podUID="55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" containerName="payload-pre-processing"
Oct 31 01:40:22.470744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.470697    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" containerName="payload-pre-processing"
Oct 31 01:40:22.470744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.470717    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1371773a-87d2-4be1-a0d0-4f5fc29e04ad" containerName="payload-processing"
Oct 31 01:40:22.470744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.470722    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1371773a-87d2-4be1-a0d0-4f5fc29e04ad" containerName="payload-processing"
Oct 31 01:40:22.470844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.470810    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="55384034-e4b0-43f0-a2a5-d6bf30c4a4ca" containerName="payload-pre-processing"
Oct 31 01:40:22.470844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.470822    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1371773a-87d2-4be1-a0d0-4f5fc29e04ad" containerName="payload-processing"
Oct 31 01:40:22.474198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.474180    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.527560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.527529    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-53af08\""
Oct 31 01:40:22.555751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.555705    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.555927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.555901    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfp6d\" (UniqueName: \"kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d\") pod \"payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.577004 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.576980    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-53af08-dockercfg-st7sf\""
Oct 31 01:40:22.579776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.579754    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"]
Oct 31 01:40:22.630557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.630530    5419 generic.go:356] "Generic (PLEG): container finished" podID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerID="467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16" exitCode=0
Oct 31 01:40:22.630680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.630571    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" event={"ID":"e9aa2ebd-853c-45dd-a273-b0aae3d34984","Type":"ContainerDied","Data":"467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"}
Oct 31 01:40:22.630941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.630928    5419 scope.go:122] "RemoveContainer" containerID="467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"
Oct 31 01:40:22.632559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.632535    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:22.632637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.632599    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:22.632675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.632634    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:22.657556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.657532    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.657662 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.657629    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfp6d\" (UniqueName: \"kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d\") pod \"payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.658400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.658378    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.716503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.716466    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfp6d\" (UniqueName: \"kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d\") pod \"payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:22.788311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:22.788218    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:23.189517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.187451    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"]
Oct 31 01:40:23.262062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.262032    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"]
Oct 31 01:40:23.267821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.267804    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.322661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.322631    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"]
Oct 31 01:40:23.371489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.371453    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5c9\" (UniqueName: \"kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9\") pod \"payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.371666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.371576    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.473125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.473095    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.473663 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.473275    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5c9\" (UniqueName: \"kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9\") pod \"payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.473943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.473917    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.529840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.529803    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5c9\" (UniqueName: \"kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9\") pod \"payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.608569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.608532    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:23.641640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.641605    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt" event={"ID":"68670ba5-7785-4c9e-bdd0-f61a916c5c01","Type":"ContainerStarted","Data":"116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b"}
Oct 31 01:40:23.641802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.641669    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt" event={"ID":"68670ba5-7785-4c9e-bdd0-f61a916c5c01","Type":"ContainerStarted","Data":"b391892eaa9822ed322a2458ffe31f5cb6bb958f40a82b3537e65c71fc25d771"}
Oct 31 01:40:23.641948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.641930    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:23.648058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.648031    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" event={"ID":"e9aa2ebd-853c-45dd-a273-b0aae3d34984","Type":"ContainerStarted","Data":"8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d"}
Oct 31 01:40:23.766108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:23.765984    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt" podStartSLOduration=1.765967269 podStartE2EDuration="1.765967269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:23.765382098 +0000 UTC m=+2145.066338675" watchObservedRunningTime="2026-10-31 01:40:23.765967269 +0000 UTC m=+2145.066923846"
Oct 31 01:40:24.021835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.021809    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"]
Oct 31 01:40:24.023978 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:40:24.023560    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765496b6_57f5_419b_bec5_93a7758586f3.slice/crio-036d4e966c49688465aab02a509f6704dc792bc6074c0f442db8008df1739c90 WatchSource:0}: Error finding container 036d4e966c49688465aab02a509f6704dc792bc6074c0f442db8008df1739c90: Status 404 returned error can't find the container with id 036d4e966c49688465aab02a509f6704dc792bc6074c0f442db8008df1739c90
Oct 31 01:40:24.506654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.506622    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:40:24.507080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.506671    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:40:24.507080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.506734    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 01:40:24.662513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.662472    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx" event={"ID":"765496b6-57f5-419b-bec5-93a7758586f3","Type":"ContainerStarted","Data":"7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33"}
Oct 31 01:40:24.662710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.662536    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx" event={"ID":"765496b6-57f5-419b-bec5-93a7758586f3","Type":"ContainerStarted","Data":"036d4e966c49688465aab02a509f6704dc792bc6074c0f442db8008df1739c90"}
Oct 31 01:40:24.662710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.662650    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:24.802241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:24.801484    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx" podStartSLOduration=1.80146153 podStartE2EDuration="1.80146153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:24.778538326 +0000 UTC m=+2146.079494917" watchObservedRunningTime="2026-10-31 01:40:24.80146153 +0000 UTC m=+2146.102418110"
Oct 31 01:40:25.506322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:25.506281    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:40:25.506535 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:25.506348    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 01:40:26.507219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:26.507183    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:40:26.507633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:26.507241    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 01:40:27.506834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:27.506794    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:40:27.506996 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:27.506849    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 01:40:28.506333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:28.506301    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 01:40:28.506822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:28.506351    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 01:40:29.511436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:29.511408    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:40:29.511851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:29.511692    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:40:29.512499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:29.512479    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:40:34.664959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:34.664929    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:40:35.671717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:35.671679    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:40:40.376761 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:40.376724    5419 scope.go:122] "RemoveContainer" containerID="4d8b16fd432a2820354d60c35dd3aba1a723c91b80ac4f2d6dfaee0e9d37cf1b"
Oct 31 01:40:40.989167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:40.989111    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:40:44.609478 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:40:44.609438    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:40:50.325692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.325657    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"]
Oct 31 01:40:50.330859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.330838    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.407607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.407565    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k49h\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.407792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.407625    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.407792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.407702    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.407891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.407815    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.407891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.407864    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.407969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.407939    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.408027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.408014    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.408071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.408035    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.408133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.408116    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.509533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509494    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k49h\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.509736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509551    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.509736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509576    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.509736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509602    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.509736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509633    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.509736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509661    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509743    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509769    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.509808    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.510054    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.510069    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.510210    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.510298    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.510669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.510642    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.512428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.512402    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.512638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.512616    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.533094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.533063    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"]
Oct 31 01:40:50.567265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.567228    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.567443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.567245    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k49h\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h\") pod \"e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") " pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:50.644870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:50.644790    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:51.077846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.077822    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"]
Oct 31 01:40:51.079724 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:40:51.079699    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce009443_007d_4c08_bc35_32e5fd66d4c7.slice/crio-6c24302cca9e40706b66b8b746dbd6494cdc7692fcf3dd621ab524b54f63a145 WatchSource:0}: Error finding container 6c24302cca9e40706b66b8b746dbd6494cdc7692fcf3dd621ab524b54f63a145: Status 404 returned error can't find the container with id 6c24302cca9e40706b66b8b746dbd6494cdc7692fcf3dd621ab524b54f63a145
Oct 31 01:40:51.082380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.082344    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:51.090082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.082798    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:51.090082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.082921    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:51.857264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.857177    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 01:40:51.857264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.857230    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 01:40:51.901539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.901504    5419 generic.go:356] "Generic (PLEG): container finished" podID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerID="c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9" exitCode=0
Oct 31 01:40:51.901728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.901578    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" event={"ID":"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc","Type":"ContainerDied","Data":"c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"}
Oct 31 01:40:51.902032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.902014    5419 scope.go:122] "RemoveContainer" containerID="c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"
Oct 31 01:40:51.903948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.903910    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:51.904096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.903998    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:51.904096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.904040    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:40:51.905280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.905245    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" event={"ID":"ce009443-007d-4c08-bc35-32e5fd66d4c7","Type":"ContainerStarted","Data":"7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341"}
Oct 31 01:40:51.905399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:51.905299    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" event={"ID":"ce009443-007d-4c08-bc35-32e5fd66d4c7","Type":"ContainerStarted","Data":"6c24302cca9e40706b66b8b746dbd6494cdc7692fcf3dd621ab524b54f63a145"}
Oct 31 01:40:52.139665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:52.139557    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podStartSLOduration=2.139541171 podStartE2EDuration="2.139541171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:52.137275201 +0000 UTC m=+2173.438231802" watchObservedRunningTime="2026-10-31 01:40:52.139541171 +0000 UTC m=+2173.440497816"
Oct 31 01:40:52.645713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:52.645669    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:52.647129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:52.647102    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:40:52.647249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:52.647171    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 01:40:52.936707 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:52.936622    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" event={"ID":"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc","Type":"ContainerStarted","Data":"7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34"}
Oct 31 01:40:53.623355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:53.623312    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:40:53.623824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:53.623792    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 01:40:53.624100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:53.624054    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 01:40:53.646008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:53.645985    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:40:53.646128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:53.646022    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 01:40:54.623594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:54.623548    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 01:40:54.624275 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:54.624240    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 01:40:54.645920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:54.645891    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:40:54.646059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:54.645939    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 01:40:55.616593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:55.616551    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 01:40:55.616932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:55.616893    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 01:40:55.645933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:55.645896    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:40:55.646336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:55.645950    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 01:40:56.616596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:56.616556    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 01:40:56.616900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:56.616870    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 01:40:56.645920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:56.645891    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Oct 31 01:40:56.646121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:56.646102    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Oct 31 01:40:57.616636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:57.616595    5419 patch_prober.go:29] interesting pod/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 01:40:57.616821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:57.616662    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 01:40:57.650399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:57.650374    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:57.650803 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:57.650780    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:57.651422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:57.651405    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:40:58.620097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:58.620066    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:40:58.620516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:58.620498    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:40:58.621083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:40:58.621068    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:41:06.683981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.683949    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"]
Oct 31 01:41:06.688228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.688209    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:06.744418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.744386    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"]
Oct 31 01:41:06.790625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.790408    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2d49b\""
Oct 31 01:41:06.790625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.790589    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2d49b-dockercfg-pcrv8\""
Oct 31 01:41:06.850164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.821055    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwsf\" (UniqueName: \"kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:06.850164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.821109    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:06.921908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.921872    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwsf\" (UniqueName: \"kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:06.922110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.921919    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:06.923715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.923687    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:06.978903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:06.978874    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwsf\" (UniqueName: \"kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf\") pod \"payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:07.002444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:07.002416    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:07.378013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:07.377989    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"]
Oct 31 01:41:07.379716 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:41:07.379689    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ae76c2_4f9c_4e77_a48b_99ece7c0e409.slice/crio-d2256b33c2beda595bf10cd8bf3794bcd26cbf2b9e7836ded505099c620cf69e WatchSource:0}: Error finding container d2256b33c2beda595bf10cd8bf3794bcd26cbf2b9e7836ded505099c620cf69e: Status 404 returned error can't find the container with id d2256b33c2beda595bf10cd8bf3794bcd26cbf2b9e7836ded505099c620cf69e
Oct 31 01:41:08.052830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:08.052794    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8" event={"ID":"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409","Type":"ContainerStarted","Data":"028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6"}
Oct 31 01:41:08.053295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:08.052876    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8" event={"ID":"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409","Type":"ContainerStarted","Data":"d2256b33c2beda595bf10cd8bf3794bcd26cbf2b9e7836ded505099c620cf69e"}
Oct 31 01:41:08.053295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:08.052932    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:08.169078 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:08.169031    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8" podStartSLOduration=2.169018486 podStartE2EDuration="2.169018486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:08.166765413 +0000 UTC m=+2189.467721990" watchObservedRunningTime="2026-10-31 01:41:08.169018486 +0000 UTC m=+2189.469975113"
Oct 31 01:41:14.583108 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:41:14.583071    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:41:19.061280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:19.061255    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:41:26.060804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:26.060772    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"]
Oct 31 01:41:26.164493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:26.164457    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"]
Oct 31 01:41:31.061519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.061458    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m" podUID="60630003-e9bf-416e-bc71-ad136e79c8cc" containerName="payload-processing" containerID="cri-o://0c8478c998d34067f394e30a63f54e5903eb4f788684dac94977b095493d4aef" gracePeriod=25
Oct 31 01:41:31.165607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.165564    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv" podUID="cf12a6f3-5ccc-429b-bb15-3660e65d091a" containerName="payload-pre-processing" containerID="cri-o://99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1" gracePeriod=25
Oct 31 01:41:31.217098 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.217065    5419 generic.go:356] "Generic (PLEG): container finished" podID="60630003-e9bf-416e-bc71-ad136e79c8cc" containerID="0c8478c998d34067f394e30a63f54e5903eb4f788684dac94977b095493d4aef" exitCode=0
Oct 31 01:41:31.217314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.217153    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m" event={"ID":"60630003-e9bf-416e-bc71-ad136e79c8cc","Type":"ContainerDied","Data":"0c8478c998d34067f394e30a63f54e5903eb4f788684dac94977b095493d4aef"}
Oct 31 01:41:31.284599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.284560    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:41:31.326236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.326173    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:41:31.390200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.390162    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume\") pod \"60630003-e9bf-416e-bc71-ad136e79c8cc\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") "
Oct 31 01:41:31.390380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.390316    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv\" (UniqueName: \"kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv\") pod \"60630003-e9bf-416e-bc71-ad136e79c8cc\" (UID: \"60630003-e9bf-416e-bc71-ad136e79c8cc\") "
Oct 31 01:41:31.390979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.390557    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume" pod "60630003-e9bf-416e-bc71-ad136e79c8cc" (UID: "60630003-e9bf-416e-bc71-ad136e79c8cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:31.390979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.390949    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60630003-e9bf-416e-bc71-ad136e79c8cc-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:31.408646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.408616    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv" pod "60630003-e9bf-416e-bc71-ad136e79c8cc" (UID: "60630003-e9bf-416e-bc71-ad136e79c8cc"). InnerVolumeSpecName "kube-api-access-fcxxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:31.427280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.427255    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:41:31.492157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.492112    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume\") pod \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") "
Oct 31 01:41:31.492303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.492270    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh\" (UniqueName: \"kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh\") pod \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\" (UID: \"cf12a6f3-5ccc-429b-bb15-3660e65d091a\") "
Oct 31 01:41:31.492511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.492490    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume" pod "cf12a6f3-5ccc-429b-bb15-3660e65d091a" (UID: "cf12a6f3-5ccc-429b-bb15-3660e65d091a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:31.492644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.492629    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxxv\" (UniqueName: \"kubernetes.io/projected/60630003-e9bf-416e-bc71-ad136e79c8cc-kube-api-access-fcxxv\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:31.492681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.492648    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf12a6f3-5ccc-429b-bb15-3660e65d091a-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:31.494594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.494572    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh" pod "cf12a6f3-5ccc-429b-bb15-3660e65d091a" (UID: "cf12a6f3-5ccc-429b-bb15-3660e65d091a"). InnerVolumeSpecName "kube-api-access-kckgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:31.593596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:31.593509    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckgh\" (UniqueName: \"kubernetes.io/projected/cf12a6f3-5ccc-429b-bb15-3660e65d091a-kube-api-access-kckgh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:32.225185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.225131    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"
Oct 31 01:41:32.225644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.225154    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m" event={"ID":"60630003-e9bf-416e-bc71-ad136e79c8cc","Type":"ContainerDied","Data":"c32ff29458d9017cc4fa165e656b31dcaff99117f57d977455060563c8baf634"}
Oct 31 01:41:32.225644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.225301    5419 scope.go:122] "RemoveContainer" containerID="0c8478c998d34067f394e30a63f54e5903eb4f788684dac94977b095493d4aef"
Oct 31 01:41:32.226816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.226571    5419 generic.go:356] "Generic (PLEG): container finished" podID="cf12a6f3-5ccc-429b-bb15-3660e65d091a" containerID="99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1" exitCode=0
Oct 31 01:41:32.226816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.226627    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"
Oct 31 01:41:32.226816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.226651    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv" event={"ID":"cf12a6f3-5ccc-429b-bb15-3660e65d091a","Type":"ContainerDied","Data":"99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1"}
Oct 31 01:41:32.226816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.226705    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv" event={"ID":"cf12a6f3-5ccc-429b-bb15-3660e65d091a","Type":"ContainerDied","Data":"e52a064d164924a3e63415ecaf62fc5e05793ba7a9e751288218a460ffdc8bd8"}
Oct 31 01:41:32.236191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.236170    5419 scope.go:122] "RemoveContainer" containerID="99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1"
Oct 31 01:41:32.245681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.245664    5419 scope.go:122] "RemoveContainer" containerID="99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1"
Oct 31 01:41:32.245966 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:41:32.245944    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1\": container with ID starting with 99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1 not found: ID does not exist" containerID="99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1"
Oct 31 01:41:32.246012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.245975    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1"} err="failed to get container status \"99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1\": rpc error: code = NotFound desc = could not find container \"99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1\": container with ID starting with 99556d621c6248743c6b349ea1e8a3b54857f994485e01bc105569add237c5d1 not found: ID does not exist"
Oct 31 01:41:32.388479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.388360    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"]
Oct 31 01:41:32.391368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.391338    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df5d0b-5795dbdff7-4x82m"]
Oct 31 01:41:32.499765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.499735    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"]
Oct 31 01:41:32.503363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.503328    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df5d0b-888475979-kzbmv"]
Oct 31 01:41:32.838360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.838332    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60630003-e9bf-416e-bc71-ad136e79c8cc" path="/var/lib/kubelet/pods/60630003-e9bf-416e-bc71-ad136e79c8cc/volumes"
Oct 31 01:41:32.838743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:32.838731    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf12a6f3-5ccc-429b-bb15-3660e65d091a" path="/var/lib/kubelet/pods/cf12a6f3-5ccc-429b-bb15-3660e65d091a/volumes"
Oct 31 01:41:39.170954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.170921    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"]
Oct 31 01:41:39.171570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.171554    5419 cpu_manager.go:414] "containerMap: removing container" podUID="60630003-e9bf-416e-bc71-ad136e79c8cc" containerName="payload-processing"
Oct 31 01:41:39.171629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.171571    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60630003-e9bf-416e-bc71-ad136e79c8cc" containerName="payload-processing"
Oct 31 01:41:39.171629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.171584    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cf12a6f3-5ccc-429b-bb15-3660e65d091a" containerName="payload-pre-processing"
Oct 31 01:41:39.171629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.171589    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf12a6f3-5ccc-429b-bb15-3660e65d091a" containerName="payload-pre-processing"
Oct 31 01:41:39.171728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.171685    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf12a6f3-5ccc-429b-bb15-3660e65d091a" containerName="payload-pre-processing"
Oct 31 01:41:39.171728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.171703    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="60630003-e9bf-416e-bc71-ad136e79c8cc" containerName="payload-processing"
Oct 31 01:41:39.176455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.176439    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.232698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.232666    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"]
Oct 31 01:41:39.277901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.277874    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:41:39.277901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.277892    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:41:39.278099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.277934    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:41:39.278099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.277998    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:41:39.280881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.280862    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"]
Oct 31 01:41:39.296288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.296264    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4mtp\" (UniqueName: \"kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.296420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.296393    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.296468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.296426    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.398009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.397977    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.398203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.398016    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.398203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.398132    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4mtp\" (UniqueName: \"kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.407680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.399248    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.408813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.408787    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.456522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.456457    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4mtp\" (UniqueName: \"kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp\") pod \"maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.490131 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.490104    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:39.892375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:39.892342    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"]
Oct 31 01:41:39.895220 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:41:39.895196    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1017518f_c4f8_4285_a45a_d448a63b47b4.slice/crio-ee69337d822b6591869fd07714d39a4412d9070343ce2692ff40d53bb8363c6c WatchSource:0}: Error finding container ee69337d822b6591869fd07714d39a4412d9070343ce2692ff40d53bb8363c6c: Status 404 returned error can't find the container with id ee69337d822b6591869fd07714d39a4412d9070343ce2692ff40d53bb8363c6c
Oct 31 01:41:40.304864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:40.304818    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5" event={"ID":"1017518f-c4f8-4285-a45a-d448a63b47b4","Type":"ContainerStarted","Data":"807281cac5628ace84b28d3489f5d3f2f6b6cf5e4b3f9f349a46d16ac127cc8e"}
Oct 31 01:41:40.305303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:40.304884    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5" event={"ID":"1017518f-c4f8-4285-a45a-d448a63b47b4","Type":"ContainerStarted","Data":"ee69337d822b6591869fd07714d39a4412d9070343ce2692ff40d53bb8363c6c"}
Oct 31 01:41:40.410528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:40.410497    5419 scope.go:122] "RemoveContainer" containerID="3f5d99168f1bd6138fd485ee8875e19cf9054c8d383a1f01db48a3b0144dd1eb"
Oct 31 01:41:40.420668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:40.420616    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5" podStartSLOduration=1.4206031829999999 podStartE2EDuration="1.420603183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:40.417602126 +0000 UTC m=+2221.718558716" watchObservedRunningTime="2026-10-31 01:41:40.420603183 +0000 UTC m=+2221.721559759"
Oct 31 01:41:41.312666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:41.312632    5419 generic.go:356] "Generic (PLEG): container finished" podID="1017518f-c4f8-4285-a45a-d448a63b47b4" containerID="807281cac5628ace84b28d3489f5d3f2f6b6cf5e4b3f9f349a46d16ac127cc8e" exitCode=0
Oct 31 01:41:41.313073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:41.312717    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5" event={"ID":"1017518f-c4f8-4285-a45a-d448a63b47b4","Type":"ContainerDied","Data":"807281cac5628ace84b28d3489f5d3f2f6b6cf5e4b3f9f349a46d16ac127cc8e"}
Oct 31 01:41:42.495103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.495075    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:42.562229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.562201    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca\") pod \"1017518f-c4f8-4285-a45a-d448a63b47b4\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") "
Oct 31 01:41:42.562402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.562307    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca\") pod \"1017518f-c4f8-4285-a45a-d448a63b47b4\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") "
Oct 31 01:41:42.562402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.562345    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp\" (UniqueName: \"kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp\") pod \"1017518f-c4f8-4285-a45a-d448a63b47b4\" (UID: \"1017518f-c4f8-4285-a45a-d448a63b47b4\") "
Oct 31 01:41:42.562694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.562671    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca" pod "1017518f-c4f8-4285-a45a-d448a63b47b4" (UID: "1017518f-c4f8-4285-a45a-d448a63b47b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:42.564616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.564590    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca" pod "1017518f-c4f8-4285-a45a-d448a63b47b4" (UID: "1017518f-c4f8-4285-a45a-d448a63b47b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:42.564710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.564672    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp" pod "1017518f-c4f8-4285-a45a-d448a63b47b4" (UID: "1017518f-c4f8-4285-a45a-d448a63b47b4"). InnerVolumeSpecName "kube-api-access-t4mtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:42.663751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.663715    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:42.663751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.663747    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1017518f-c4f8-4285-a45a-d448a63b47b4-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:42.663751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:42.663758    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4mtp\" (UniqueName: \"kubernetes.io/projected/1017518f-c4f8-4285-a45a-d448a63b47b4-kube-api-access-t4mtp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:43.327502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.327473    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"
Oct 31 01:41:43.327706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.327508    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5" event={"ID":"1017518f-c4f8-4285-a45a-d448a63b47b4","Type":"ContainerDied","Data":"ee69337d822b6591869fd07714d39a4412d9070343ce2692ff40d53bb8363c6c"}
Oct 31 01:41:43.327706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.327563    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee69337d822b6591869fd07714d39a4412d9070343ce2692ff40d53bb8363c6c"
Oct 31 01:41:43.790381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.790345    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"]
Oct 31 01:41:43.790969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.790955    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1017518f-c4f8-4285-a45a-d448a63b47b4" containerName="revoke-keys"
Oct 31 01:41:43.791014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.790970    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1017518f-c4f8-4285-a45a-d448a63b47b4" containerName="revoke-keys"
Oct 31 01:41:43.791076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.791067    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1017518f-c4f8-4285-a45a-d448a63b47b4" containerName="revoke-keys"
Oct 31 01:41:43.794590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.794572    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.848216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.848188    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:41:43.880565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.880522    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s97mc\" (UniqueName: \"kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.880740 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.880581    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.880873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.880844    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.897177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.897128    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:41:43.897310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.897193    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:41:43.897310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.897194    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:41:43.899954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.899934    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"]
Oct 31 01:41:43.982330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.982299    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.982492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.982409    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s97mc\" (UniqueName: \"kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.982492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.982462    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.982971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.982952    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:43.985043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:43.985017    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:44.038500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.038469    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s97mc\" (UniqueName: \"kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc\") pod \"maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:44.108612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.108521    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:44.281683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.281644    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8" podUID="ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" containerName="payload-processing" containerID="cri-o://8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616" gracePeriod=25
Oct 31 01:41:44.485540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.485509    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"]
Oct 31 01:41:44.486755 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:41:44.486726    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38672beb_8e1f_471c_ac15_ac4d7297b103.slice/crio-75ef74daf439fbb689fc3eed302ad5435df141df024f7a1823111e474e05ddc3 WatchSource:0}: Error finding container 75ef74daf439fbb689fc3eed302ad5435df141df024f7a1823111e474e05ddc3: Status 404 returned error can't find the container with id 75ef74daf439fbb689fc3eed302ad5435df141df024f7a1823111e474e05ddc3
Oct 31 01:41:44.542947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.542925    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:41:44.581990 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:41:44.581956    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:41:44.689281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.689215    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume\") pod \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") "
Oct 31 01:41:44.689484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.689464    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw\" (UniqueName: \"kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw\") pod \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\" (UID: \"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76\") "
Oct 31 01:41:44.689584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.689557    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume" pod "ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" (UID: "ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:44.689967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.689944    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:44.691631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.691608    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw" pod "ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" (UID: "ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76"). InnerVolumeSpecName "kube-api-access-vdbkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:44.791636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:44.791611    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdbkw\" (UniqueName: \"kubernetes.io/projected/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76-kube-api-access-vdbkw\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:45.340619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.340517    5419 generic.go:356] "Generic (PLEG): container finished" podID="ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" containerID="8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616" exitCode=0
Oct 31 01:41:45.340619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.340568    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"
Oct 31 01:41:45.340858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.340610    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8" event={"ID":"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76","Type":"ContainerDied","Data":"8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616"}
Oct 31 01:41:45.340858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.340667    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8" event={"ID":"ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76","Type":"ContainerDied","Data":"ab25f095bd64a602a7059b360de2b2bdea312215122cc3288a8f83cbc6fe2e2d"}
Oct 31 01:41:45.340858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.340695    5419 scope.go:122] "RemoveContainer" containerID="8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616"
Oct 31 01:41:45.342821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.342780    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz" event={"ID":"38672beb-8e1f-471c-ac15-ac4d7297b103","Type":"ContainerStarted","Data":"5faa4fca6978ac35c3188e9c1fbc4a2e9bdd7d8e4f4a6f2b6f715773f265c45c"}
Oct 31 01:41:45.342979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.342835    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz" event={"ID":"38672beb-8e1f-471c-ac15-ac4d7297b103","Type":"ContainerStarted","Data":"75ef74daf439fbb689fc3eed302ad5435df141df024f7a1823111e474e05ddc3"}
Oct 31 01:41:45.351188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.351166    5419 scope.go:122] "RemoveContainer" containerID="8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616"
Oct 31 01:41:45.351484 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:41:45.351466    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616\": container with ID starting with 8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616 not found: ID does not exist" containerID="8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616"
Oct 31 01:41:45.351545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.351495    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616"} err="failed to get container status \"8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616\": rpc error: code = NotFound desc = could not find container \"8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616\": container with ID starting with 8de5b8da2e24843c929a2fe57f6a6a6b1ab0a0c05ff732b775c84745f91f2616 not found: ID does not exist"
Oct 31 01:41:45.502657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.502614    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"]
Oct 31 01:41:45.505973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.505935    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-96d6c7797-tkjx8"]
Oct 31 01:41:45.616899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:45.616798    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz" podStartSLOduration=2.616783483 podStartE2EDuration="2.616783483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:45.615463715 +0000 UTC m=+2226.916420306" watchObservedRunningTime="2026-10-31 01:41:45.616783483 +0000 UTC m=+2226.917740059"
Oct 31 01:41:46.355886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:46.355839    5419 generic.go:356] "Generic (PLEG): container finished" podID="38672beb-8e1f-471c-ac15-ac4d7297b103" containerID="5faa4fca6978ac35c3188e9c1fbc4a2e9bdd7d8e4f4a6f2b6f715773f265c45c" exitCode=0
Oct 31 01:41:46.356318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:46.355927    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz" event={"ID":"38672beb-8e1f-471c-ac15-ac4d7297b103","Type":"ContainerDied","Data":"5faa4fca6978ac35c3188e9c1fbc4a2e9bdd7d8e4f4a6f2b6f715773f265c45c"}
Oct 31 01:41:46.838091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:46.838057    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" path="/var/lib/kubelet/pods/ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76/volumes"
Oct 31 01:41:47.523958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.523933    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:47.626941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.626864    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca\") pod \"38672beb-8e1f-471c-ac15-ac4d7297b103\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") "
Oct 31 01:41:47.626941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.626905    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc\" (UniqueName: \"kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc\") pod \"38672beb-8e1f-471c-ac15-ac4d7297b103\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") "
Oct 31 01:41:47.627175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.627133    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca\") pod \"38672beb-8e1f-471c-ac15-ac4d7297b103\" (UID: \"38672beb-8e1f-471c-ac15-ac4d7297b103\") "
Oct 31 01:41:47.627521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.627482    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca" pod "38672beb-8e1f-471c-ac15-ac4d7297b103" (UID: "38672beb-8e1f-471c-ac15-ac4d7297b103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:41:47.627812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.627793    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:47.629189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.629160    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca" pod "38672beb-8e1f-471c-ac15-ac4d7297b103" (UID: "38672beb-8e1f-471c-ac15-ac4d7297b103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:41:47.629304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.629264    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc" pod "38672beb-8e1f-471c-ac15-ac4d7297b103" (UID: "38672beb-8e1f-471c-ac15-ac4d7297b103"). InnerVolumeSpecName "kube-api-access-s97mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:41:47.729099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.729070    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38672beb-8e1f-471c-ac15-ac4d7297b103-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:47.729099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:47.729098    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s97mc\" (UniqueName: \"kubernetes.io/projected/38672beb-8e1f-471c-ac15-ac4d7297b103-kube-api-access-s97mc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:41:48.369846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:48.369819    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"
Oct 31 01:41:48.370008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:48.369815    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz" event={"ID":"38672beb-8e1f-471c-ac15-ac4d7297b103","Type":"ContainerDied","Data":"75ef74daf439fbb689fc3eed302ad5435df141df024f7a1823111e474e05ddc3"}
Oct 31 01:41:48.370008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:48.369940    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ef74daf439fbb689fc3eed302ad5435df141df024f7a1823111e474e05ddc3"
Oct 31 01:41:59.336988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:59.336953    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"]
Oct 31 01:41:59.337624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:59.337172    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" containerID="cri-o://5aa53c8e1427b19c19c0255fff58f7046590931c8be268b9c053510b809b1381" gracePeriod=30
Oct 31 01:41:59.990111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:59.990073    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:41:59.990304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:41:59.990189    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:42:04.475981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.475947    5419 generic.go:356] "Generic (PLEG): container finished" podID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerID="5aa53c8e1427b19c19c0255fff58f7046590931c8be268b9c053510b809b1381" exitCode=0
Oct 31 01:42:04.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.476024    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" event={"ID":"e92093e7-11b5-4988-ab07-ed138ce067f1","Type":"ContainerDied","Data":"5aa53c8e1427b19c19c0255fff58f7046590931c8be268b9c053510b809b1381"}
Oct 31 01:42:04.606204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.606112    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:42:04.724251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724214    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724265    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724298    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724326    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724372    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724403    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724633    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724682    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:04.724735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724658    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:04.724735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724685    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.724735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724695    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:04.724961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724813    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:04.724961 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.724918    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs\") pod \"e92093e7-11b5-4988-ab07-ed138ce067f1\" (UID: \"e92093e7-11b5-4988-ab07-ed138ce067f1\") "
Oct 31 01:42:04.725403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.725276    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:04.725694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.725675    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.725737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.725703    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.725737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.725731    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.725813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.725746    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.725813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.725761    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e92093e7-11b5-4988-ab07-ed138ce067f1-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.726725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.726694    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:42:04.726925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.726905    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "kube-api-access-nmzrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:04.727363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.727343    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:04.727460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.727384    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token" pod "e92093e7-11b5-4988-ab07-ed138ce067f1" (UID: "e92093e7-11b5-4988-ab07-ed138ce067f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:04.826612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.826575    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.826612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.826604    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmzrf\" (UniqueName: \"kubernetes.io/projected/e92093e7-11b5-4988-ab07-ed138ce067f1-kube-api-access-nmzrf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.826612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.826615    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:04.826865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:04.826623    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e92093e7-11b5-4988-ab07-ed138ce067f1-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:05.484102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:05.484071    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh" event={"ID":"e92093e7-11b5-4988-ab07-ed138ce067f1","Type":"ContainerDied","Data":"6daca443959f65a89a94c103dc1c3a8b60d1b43f535bef465cc7e5eb17ac6081"}
Oct 31 01:42:05.484586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:05.484127    5419 scope.go:122] "RemoveContainer" containerID="5aa53c8e1427b19c19c0255fff58f7046590931c8be268b9c053510b809b1381"
Oct 31 01:42:05.484586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:05.484078    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"
Oct 31 01:42:05.651765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:05.650863    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"]
Oct 31 01:42:05.697045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:05.697009    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-tw6gh"]
Oct 31 01:42:06.838384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:06.838354    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" path="/var/lib/kubelet/pods/e92093e7-11b5-4988-ab07-ed138ce067f1/volumes"
Oct 31 01:42:14.583301 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:14.583268    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:42:33.281909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:33.281832    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:42:36.246875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.246849    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"]
Oct 31 01:42:36.247351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.247055    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing" containerID="cri-o://1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689" gracePeriod=30
Oct 31 01:42:36.351770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.351732    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"]
Oct 31 01:42:36.352027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.351995    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t" podUID="cc34e482-597e-4030-8d0f-bcd06daceede" containerName="payload-pre-processing" containerID="cri-o://36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb" gracePeriod=30
Oct 31 01:42:36.542894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.542865    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:42:36.581893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.581865    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq\" (UniqueName: \"kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq\") pod \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") "
Oct 31 01:42:36.582085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.581979    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume\") pod \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\" (UID: \"38a4f2d4-9a54-471f-8f4e-16bee6db1d09\") "
Oct 31 01:42:36.582451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.582421    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume" pod "38a4f2d4-9a54-471f-8f4e-16bee6db1d09" (UID: "38a4f2d4-9a54-471f-8f4e-16bee6db1d09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:36.582659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.582605    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.584375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.584346    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq" pod "38a4f2d4-9a54-471f-8f4e-16bee6db1d09" (UID: "38a4f2d4-9a54-471f-8f4e-16bee6db1d09"). InnerVolumeSpecName "kube-api-access-4d9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:36.603033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.603012    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:42:36.683299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.683263    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2\" (UniqueName: \"kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2\") pod \"cc34e482-597e-4030-8d0f-bcd06daceede\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") "
Oct 31 01:42:36.683469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.683427    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume\") pod \"cc34e482-597e-4030-8d0f-bcd06daceede\" (UID: \"cc34e482-597e-4030-8d0f-bcd06daceede\") "
Oct 31 01:42:36.683844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.683812    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume" pod "cc34e482-597e-4030-8d0f-bcd06daceede" (UID: "cc34e482-597e-4030-8d0f-bcd06daceede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:36.684264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.684242    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d9rq\" (UniqueName: \"kubernetes.io/projected/38a4f2d4-9a54-471f-8f4e-16bee6db1d09-kube-api-access-4d9rq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.684348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.684269    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc34e482-597e-4030-8d0f-bcd06daceede-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.685760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.685735    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2" pod "cc34e482-597e-4030-8d0f-bcd06daceede" (UID: "cc34e482-597e-4030-8d0f-bcd06daceede"). InnerVolumeSpecName "kube-api-access-whwq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:36.728701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.728670    5419 generic.go:356] "Generic (PLEG): container finished" podID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerID="1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689" exitCode=0
Oct 31 01:42:36.728829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.728721    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"
Oct 31 01:42:36.728829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.728763    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" event={"ID":"38a4f2d4-9a54-471f-8f4e-16bee6db1d09","Type":"ContainerDied","Data":"1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689"}
Oct 31 01:42:36.728829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.728819    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f" event={"ID":"38a4f2d4-9a54-471f-8f4e-16bee6db1d09","Type":"ContainerDied","Data":"202afd4e5f672c24fc04878d90adade9a6cf76b22e33ac714543e905226c8718"}
Oct 31 01:42:36.728956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.728848    5419 scope.go:122] "RemoveContainer" containerID="1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689"
Oct 31 01:42:36.730442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.730417    5419 generic.go:356] "Generic (PLEG): container finished" podID="cc34e482-597e-4030-8d0f-bcd06daceede" containerID="36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb" exitCode=0
Oct 31 01:42:36.730575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.730490    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"
Oct 31 01:42:36.730575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.730499    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t" event={"ID":"cc34e482-597e-4030-8d0f-bcd06daceede","Type":"ContainerDied","Data":"36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb"}
Oct 31 01:42:36.730699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.730600    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t" event={"ID":"cc34e482-597e-4030-8d0f-bcd06daceede","Type":"ContainerDied","Data":"0442f5d8608764d22fda7bbe35be0c8877991521e8cee08bb57512c074773604"}
Oct 31 01:42:36.739740 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.739717    5419 scope.go:122] "RemoveContainer" containerID="14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"
Oct 31 01:42:36.748877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.748859    5419 scope.go:122] "RemoveContainer" containerID="1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689"
Oct 31 01:42:36.749168 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:36.749120    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689\": container with ID starting with 1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689 not found: ID does not exist" containerID="1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689"
Oct 31 01:42:36.749265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.749172    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689"} err="failed to get container status \"1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689\": rpc error: code = NotFound desc = could not find container \"1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689\": container with ID starting with 1fa406a4f22bf82eec416897bd58c38415445d6310bb661244c354cea552c689 not found: ID does not exist"
Oct 31 01:42:36.749265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.749191    5419 scope.go:122] "RemoveContainer" containerID="14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"
Oct 31 01:42:36.749431 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:36.749415    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238\": container with ID starting with 14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238 not found: ID does not exist" containerID="14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"
Oct 31 01:42:36.749473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.749440    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238"} err="failed to get container status \"14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238\": rpc error: code = NotFound desc = could not find container \"14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238\": container with ID starting with 14a7543578871dce5cbab13446715bfe1aa6c5cca713f8b066bbda3e226fc238 not found: ID does not exist"
Oct 31 01:42:36.749473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.749455    5419 scope.go:122] "RemoveContainer" containerID="36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb"
Oct 31 01:42:36.758628 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.758607    5419 scope.go:122] "RemoveContainer" containerID="36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb"
Oct 31 01:42:36.758945 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:36.758925    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb\": container with ID starting with 36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb not found: ID does not exist" containerID="36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb"
Oct 31 01:42:36.759006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.758955    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb"} err="failed to get container status \"36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb\": rpc error: code = NotFound desc = could not find container \"36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb\": container with ID starting with 36b130cdd0ac8878b6d3262adf7cc5c139cf0f3cb16dcc458b5a432dd94000bb not found: ID does not exist"
Oct 31 01:42:36.785914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.785886    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whwq2\" (UniqueName: \"kubernetes.io/projected/cc34e482-597e-4030-8d0f-bcd06daceede-kube-api-access-whwq2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:36.909194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.908253    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"]
Oct 31 01:42:36.909194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:36.908301    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-99c8bd-8c75c7944-76r8f"]
Oct 31 01:42:37.004446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:37.004406    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"]
Oct 31 01:42:37.013201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:37.013104    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-99c8bd-b945cf7f9-87s2t"]
Oct 31 01:42:38.141666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:38.141630    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"]
Oct 31 01:42:38.190388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:38.190358    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-99c8bd-ptms5"]
Oct 31 01:42:38.838633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:38.838601    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1017518f-c4f8-4285-a45a-d448a63b47b4" path="/var/lib/kubelet/pods/1017518f-c4f8-4285-a45a-d448a63b47b4/volumes"
Oct 31 01:42:38.839000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:38.838986    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" path="/var/lib/kubelet/pods/38a4f2d4-9a54-471f-8f4e-16bee6db1d09/volumes"
Oct 31 01:42:38.839400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:38.839386    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc34e482-597e-4030-8d0f-bcd06daceede" path="/var/lib/kubelet/pods/cc34e482-597e-4030-8d0f-bcd06daceede/volumes"
Oct 31 01:42:44.583370 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:44.583334    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:42:44.939081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:44.939011    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"]
Oct 31 01:42:44.939276 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:44.939249    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy" containerID="cri-o://7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280" gracePeriod=30
Oct 31 01:42:45.817510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.817472    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"]
Oct 31 01:42:45.818015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818002    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" containerName="payload-processing"
Oct 31 01:42:45.818072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818016    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" containerName="payload-processing"
Oct 31 01:42:45.818072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818047    5419 cpu_manager.go:414] "containerMap: removing container" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing"
Oct 31 01:42:45.818072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818052    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing"
Oct 31 01:42:45.818072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818063    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy"
Oct 31 01:42:45.818072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818069    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818079    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cc34e482-597e-4030-8d0f-bcd06daceede" containerName="payload-pre-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818084    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc34e482-597e-4030-8d0f-bcd06daceede" containerName="payload-pre-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818092    5419 cpu_manager.go:414] "containerMap: removing container" podUID="38672beb-8e1f-471c-ac15-ac4d7297b103" containerName="revoke-keys"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818096    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38672beb-8e1f-471c-ac15-ac4d7297b103" containerName="revoke-keys"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818107    5419 cpu_manager.go:414] "containerMap: removing container" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818111    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818214    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc34e482-597e-4030-8d0f-bcd06daceede" containerName="payload-pre-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818225    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="38672beb-8e1f-471c-ac15-ac4d7297b103" containerName="revoke-keys"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818234    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818241    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4d041f-2f27-4b12-bfc5-bb7b3bea2b76" containerName="payload-processing"
Oct 31 01:42:45.818259 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.818253    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92093e7-11b5-4988-ab07-ed138ce067f1" containerName="istio-proxy"
Oct 31 01:42:45.823129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.823110    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:45.875110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.875078    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:42:45.918302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.918270    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:45.918461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.918327    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:45.918502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.918469    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kplqb\" (UniqueName: \"kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:45.924500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.924475    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:42:45.924500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.924485    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:42:45.924705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.924556    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:42:45.927508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:45.927485    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"]
Oct 31 01:42:46.032346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.032294    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.032563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.032436    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.032688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.032658    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kplqb\" (UniqueName: \"kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.039456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.039425    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.044401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.044372    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.089481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.089403    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kplqb\" (UniqueName: \"kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb\") pod \"maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.143447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.143407    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:46.519943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.519912    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"]
Oct 31 01:42:46.521771 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:42:46.521742    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f156dc_509e_4c9b_8273_f57aec02449c.slice/crio-ec9eb1fd281b08f7f8c05745d4a9d8af899dc10ee8fe7840b6ccb5cae5d0e09e WatchSource:0}: Error finding container ec9eb1fd281b08f7f8c05745d4a9d8af899dc10ee8fe7840b6ccb5cae5d0e09e: Status 404 returned error can't find the container with id ec9eb1fd281b08f7f8c05745d4a9d8af899dc10ee8fe7840b6ccb5cae5d0e09e
Oct 31 01:42:46.794204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.794173    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj" event={"ID":"e4f156dc-509e-4c9b-8273-f57aec02449c","Type":"ContainerStarted","Data":"f71cef98421db23a50e3e62865d5eac77a540a2e6c7d888fd74696a3c8b40c80"}
Oct 31 01:42:46.794329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.794228    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj" event={"ID":"e4f156dc-509e-4c9b-8273-f57aec02449c","Type":"ContainerStarted","Data":"ec9eb1fd281b08f7f8c05745d4a9d8af899dc10ee8fe7840b6ccb5cae5d0e09e"}
Oct 31 01:42:46.909758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:46.909703    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj" podStartSLOduration=1.909688045 podStartE2EDuration="1.909688045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:46.90680807 +0000 UTC m=+2288.207764647" watchObservedRunningTime="2026-10-31 01:42:46.909688045 +0000 UTC m=+2288.210644613"
Oct 31 01:42:47.802792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:47.802758    5419 generic.go:356] "Generic (PLEG): container finished" podID="e4f156dc-509e-4c9b-8273-f57aec02449c" containerID="f71cef98421db23a50e3e62865d5eac77a540a2e6c7d888fd74696a3c8b40c80" exitCode=0
Oct 31 01:42:47.802989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:47.802820    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj" event={"ID":"e4f156dc-509e-4c9b-8273-f57aec02449c","Type":"ContainerDied","Data":"f71cef98421db23a50e3e62865d5eac77a540a2e6c7d888fd74696a3c8b40c80"}
Oct 31 01:42:48.953644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:48.953622    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:49.068659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.068574    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca\") pod \"e4f156dc-509e-4c9b-8273-f57aec02449c\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") "
Oct 31 01:42:49.069333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.068669    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb\" (UniqueName: \"kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb\") pod \"e4f156dc-509e-4c9b-8273-f57aec02449c\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") "
Oct 31 01:42:49.069333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.068898    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca\") pod \"e4f156dc-509e-4c9b-8273-f57aec02449c\" (UID: \"e4f156dc-509e-4c9b-8273-f57aec02449c\") "
Oct 31 01:42:49.069521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.069465    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca" pod "e4f156dc-509e-4c9b-8273-f57aec02449c" (UID: "e4f156dc-509e-4c9b-8273-f57aec02449c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:49.071664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.071621    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb" pod "e4f156dc-509e-4c9b-8273-f57aec02449c" (UID: "e4f156dc-509e-4c9b-8273-f57aec02449c"). InnerVolumeSpecName "kube-api-access-kplqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:49.071793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.071765    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca" pod "e4f156dc-509e-4c9b-8273-f57aec02449c" (UID: "e4f156dc-509e-4c9b-8273-f57aec02449c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:42:49.170219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.170191    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:49.170219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.170217    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f156dc-509e-4c9b-8273-f57aec02449c-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:49.170419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.170226    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kplqb\" (UniqueName: \"kubernetes.io/projected/e4f156dc-509e-4c9b-8273-f57aec02449c-kube-api-access-kplqb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:49.817673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.817645    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"
Oct 31 01:42:49.817673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.817666    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj" event={"ID":"e4f156dc-509e-4c9b-8273-f57aec02449c","Type":"ContainerDied","Data":"ec9eb1fd281b08f7f8c05745d4a9d8af899dc10ee8fe7840b6ccb5cae5d0e09e"}
Oct 31 01:42:49.817953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:49.817746    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec9eb1fd281b08f7f8c05745d4a9d8af899dc10ee8fe7840b6ccb5cae5d0e09e"
Oct 31 01:42:50.208128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.208104    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:42:50.283640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.283609    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.283809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.283656    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.283809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.283727    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.283809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.283772    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.283947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.283897    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.283947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.283931    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.284045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284001    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.284109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284042    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.284109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284073    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5\") pod \"aa5b7129-09bc-45e9-b97a-5759d316f60f\" (UID: \"aa5b7129-09bc-45e9-b97a-5759d316f60f\") "
Oct 31 01:42:50.284109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284073    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:50.284309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284234    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:50.284738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284466    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:42:50.284738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284585    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:50.284738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284639    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:50.285020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284934    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa5b7129-09bc-45e9-b97a-5759d316f60f-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.285020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284958    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.285020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284972    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.285020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284984    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.285020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.284997    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.286124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.286090    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:42:50.286265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.286182    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:42:50.286265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.286215    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:50.286399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.286382    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5" pod "aa5b7129-09bc-45e9-b97a-5759d316f60f" (UID: "aa5b7129-09bc-45e9-b97a-5759d316f60f"). InnerVolumeSpecName "kube-api-access-lvdb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:42:50.386967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.386890    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.386967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.386929    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.386967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.386944    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa5b7129-09bc-45e9-b97a-5759d316f60f-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.386967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.386957    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdb5\" (UniqueName: \"kubernetes.io/projected/aa5b7129-09bc-45e9-b97a-5759d316f60f-kube-api-access-lvdb5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:42:50.825690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.825658    5419 generic.go:356] "Generic (PLEG): container finished" podID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerID="7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280" exitCode=0
Oct 31 01:42:50.825875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.825709    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"
Oct 31 01:42:50.825875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.825737    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" event={"ID":"aa5b7129-09bc-45e9-b97a-5759d316f60f","Type":"ContainerDied","Data":"7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280"}
Oct 31 01:42:50.825875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.825791    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd" event={"ID":"aa5b7129-09bc-45e9-b97a-5759d316f60f","Type":"ContainerDied","Data":"52b0a523f4ba196235d94f52058a56e25490b0860fe71802f50a0a0a63b7252c"}
Oct 31 01:42:50.825875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.825815    5419 scope.go:122] "RemoveContainer" containerID="7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280"
Oct 31 01:42:50.837207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.837182    5419 scope.go:122] "RemoveContainer" containerID="e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"
Oct 31 01:42:50.846317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.846301    5419 scope.go:122] "RemoveContainer" containerID="7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280"
Oct 31 01:42:50.846565 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:50.846549    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280\": container with ID starting with 7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280 not found: ID does not exist" containerID="7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280"
Oct 31 01:42:50.846629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.846572    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280"} err="failed to get container status \"7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280\": rpc error: code = NotFound desc = could not find container \"7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280\": container with ID starting with 7a6d3eded96ec632b5ea693856087a2fe582ad878f44d15dc9acd1e4e74ad280 not found: ID does not exist"
Oct 31 01:42:50.846629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.846591    5419 scope.go:122] "RemoveContainer" containerID="e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"
Oct 31 01:42:50.846853 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:42:50.846835    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca\": container with ID starting with e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca not found: ID does not exist" containerID="e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"
Oct 31 01:42:50.846894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.846860    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca"} err="failed to get container status \"e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca\": rpc error: code = NotFound desc = could not find container \"e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca\": container with ID starting with e532b576866860462501bea38f7f403b738f4f77d2ac0ff957c02169d4c146ca not found: ID does not exist"
Oct 31 01:42:50.990444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:50.990414    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"]
Oct 31 01:42:51.040021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:51.039987    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-99c8bd-openshift-default-79b8766c89-nf4vd"]
Oct 31 01:42:52.838241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:42:52.838205    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" path="/var/lib/kubelet/pods/aa5b7129-09bc-45e9-b97a-5759d316f60f/volumes"
Oct 31 01:43:00.692910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.692876    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"]
Oct 31 01:43:00.693476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693460    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e4f156dc-509e-4c9b-8273-f57aec02449c" containerName="revoke-keys"
Oct 31 01:43:00.693521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693478    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f156dc-509e-4c9b-8273-f57aec02449c" containerName="revoke-keys"
Oct 31 01:43:00.693521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693493    5419 cpu_manager.go:414] "containerMap: removing container" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy"
Oct 31 01:43:00.693521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693498    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy"
Oct 31 01:43:00.693637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693626    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy"
Oct 31 01:43:00.693670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693638    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f156dc-509e-4c9b-8273-f57aec02449c" containerName="revoke-keys"
Oct 31 01:43:00.693670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693644    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy"
Oct 31 01:43:00.693670 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.693657    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a4f2d4-9a54-471f-8f4e-16bee6db1d09" containerName="payload-processing"
Oct 31 01:43:00.698502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.698480    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.753122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.753091    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"]
Oct 31 01:43:00.799130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.799103    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:00.799130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.799109    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:00.799360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.799130    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:00.799611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.799598    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:43:00.830938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.830915    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.831076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.831008    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.831076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.831035    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gf4c\" (UniqueName: \"kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.878615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.878584    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"]
Oct 31 01:43:00.879149 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.879125    5419 cpu_manager.go:414] "containerMap: removing container" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy"
Oct 31 01:43:00.879192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.879152    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa5b7129-09bc-45e9-b97a-5759d316f60f" containerName="istio-proxy"
Oct 31 01:43:00.882574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.882560    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:00.932804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.932780    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.932953 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.932811    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gf4c\" (UniqueName: \"kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.933712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.933489    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.933712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.933543    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.935933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.935911    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:00.939341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.939318    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8af859-dockercfg-gtjth\""
Oct 31 01:43:00.939454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.939361    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8af859\""
Oct 31 01:43:00.942059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.942037    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"]
Oct 31 01:43:00.989526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:00.989499    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gf4c\" (UniqueName: \"kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c\") pod \"maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:01.011936 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.011872    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:01.035300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.035273    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.035415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.035386    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdhx\" (UniqueName: \"kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx\") pod \"payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.136298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.136257    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.136453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.136347    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdhx\" (UniqueName: \"kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx\") pod \"payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.136925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.136905    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.193221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.193185    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdhx\" (UniqueName: \"kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx\") pod \"payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.196898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.196871    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.389490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.389468    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"]
Oct 31 01:43:01.390669 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:01.390638    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf912534e_9398_4d23_a85c_446a048e680e.slice/crio-2b631b3eb7165c7bf2c3030e1d2b2e36e81a37a951fdf66f585e164000523182 WatchSource:0}: Error finding container 2b631b3eb7165c7bf2c3030e1d2b2e36e81a37a951fdf66f585e164000523182: Status 404 returned error can't find the container with id 2b631b3eb7165c7bf2c3030e1d2b2e36e81a37a951fdf66f585e164000523182
Oct 31 01:43:01.572683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.572658    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"]
Oct 31 01:43:01.574265 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:01.574238    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4a5753_43ca_4f56_96c1_bb7d84f2ba9e.slice/crio-b24b42b1a03c59dd3741d87c5153d88a827547e6506b27b34dda802878517826 WatchSource:0}: Error finding container b24b42b1a03c59dd3741d87c5153d88a827547e6506b27b34dda802878517826: Status 404 returned error can't find the container with id b24b42b1a03c59dd3741d87c5153d88a827547e6506b27b34dda802878517826
Oct 31 01:43:01.675559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.675529    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"]
Oct 31 01:43:01.679884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.679864    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:01.736117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.736088    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"]
Oct 31 01:43:01.844562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.844530    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:01.844736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.844600    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkp8\" (UniqueName: \"kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8\") pod \"payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:01.897563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.897521    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" event={"ID":"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e","Type":"ContainerStarted","Data":"e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b"}
Oct 31 01:43:01.897728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.897621    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" event={"ID":"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e","Type":"ContainerStarted","Data":"b24b42b1a03c59dd3741d87c5153d88a827547e6506b27b34dda802878517826"}
Oct 31 01:43:01.897728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.897724    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:01.899114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.899089    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q" event={"ID":"f912534e-9398-4d23-a85c-446a048e680e","Type":"ContainerStarted","Data":"25e0441b981b6f086c941e5f68b9016c1f6b0e97dae085e35b6950a52996d430"}
Oct 31 01:43:01.899251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.899151    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q" event={"ID":"f912534e-9398-4d23-a85c-446a048e680e","Type":"ContainerStarted","Data":"2b631b3eb7165c7bf2c3030e1d2b2e36e81a37a951fdf66f585e164000523182"}
Oct 31 01:43:01.945927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.945863    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:01.946052 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.945972    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkp8\" (UniqueName: \"kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8\") pod \"payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:01.946607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:01.946588    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:02.002408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.002381    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkp8\" (UniqueName: \"kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8\") pod \"payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:02.128518 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.127984    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" podStartSLOduration=2.127966589 podStartE2EDuration="2.127966589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:02.010247499 +0000 UTC m=+2303.311204086" watchObservedRunningTime="2026-10-31 01:43:02.127966589 +0000 UTC m=+2303.428923168"
Oct 31 01:43:02.147220 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.147127    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q" podStartSLOduration=2.147102593 podStartE2EDuration="2.147102593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:02.127602894 +0000 UTC m=+2303.428559474" watchObservedRunningTime="2026-10-31 01:43:02.147102593 +0000 UTC m=+2303.448059172"
Oct 31 01:43:02.295035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.295002    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:02.687721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.687692    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"]
Oct 31 01:43:02.689475 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:02.689449    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ddc321_5806_498a_9ea4_38788c7d0c17.slice/crio-d9fd92d1db8da6d49e126c6042a833d85696b47edc1ab321c254acdee0d7dc31 WatchSource:0}: Error finding container d9fd92d1db8da6d49e126c6042a833d85696b47edc1ab321c254acdee0d7dc31: Status 404 returned error can't find the container with id d9fd92d1db8da6d49e126c6042a833d85696b47edc1ab321c254acdee0d7dc31
Oct 31 01:43:02.908404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.908369    5419 generic.go:356] "Generic (PLEG): container finished" podID="f912534e-9398-4d23-a85c-446a048e680e" containerID="25e0441b981b6f086c941e5f68b9016c1f6b0e97dae085e35b6950a52996d430" exitCode=0
Oct 31 01:43:02.908859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.908452    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q" event={"ID":"f912534e-9398-4d23-a85c-446a048e680e","Type":"ContainerDied","Data":"25e0441b981b6f086c941e5f68b9016c1f6b0e97dae085e35b6950a52996d430"}
Oct 31 01:43:02.910182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.910126    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq" event={"ID":"21ddc321-5806-498a-9ea4-38788c7d0c17","Type":"ContainerStarted","Data":"6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490"}
Oct 31 01:43:02.910320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.910190    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq" event={"ID":"21ddc321-5806-498a-9ea4-38788c7d0c17","Type":"ContainerStarted","Data":"d9fd92d1db8da6d49e126c6042a833d85696b47edc1ab321c254acdee0d7dc31"}
Oct 31 01:43:02.910377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:02.910315    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:03.133838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:03.133787    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq" podStartSLOduration=2.133770698 podStartE2EDuration="2.133770698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:03.133359767 +0000 UTC m=+2304.434316345" watchObservedRunningTime="2026-10-31 01:43:03.133770698 +0000 UTC m=+2304.434727275"
Oct 31 01:43:04.071065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.071036    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:04.176088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.175995    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca\") pod \"f912534e-9398-4d23-a85c-446a048e680e\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") "
Oct 31 01:43:04.176427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.176100    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca\") pod \"f912534e-9398-4d23-a85c-446a048e680e\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") "
Oct 31 01:43:04.176427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.176284    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c\" (UniqueName: \"kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c\") pod \"f912534e-9398-4d23-a85c-446a048e680e\" (UID: \"f912534e-9398-4d23-a85c-446a048e680e\") "
Oct 31 01:43:04.176562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.176505    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca" pod "f912534e-9398-4d23-a85c-446a048e680e" (UID: "f912534e-9398-4d23-a85c-446a048e680e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:04.176737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.176717    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f912534e-9398-4d23-a85c-446a048e680e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:04.178372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.178347    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca" pod "f912534e-9398-4d23-a85c-446a048e680e" (UID: "f912534e-9398-4d23-a85c-446a048e680e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:04.178507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.178456    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c" pod "f912534e-9398-4d23-a85c-446a048e680e" (UID: "f912534e-9398-4d23-a85c-446a048e680e"). InnerVolumeSpecName "kube-api-access-6gf4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:04.278016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.277984    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f912534e-9398-4d23-a85c-446a048e680e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:04.278016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.278016    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gf4c\" (UniqueName: \"kubernetes.io/projected/f912534e-9398-4d23-a85c-446a048e680e-kube-api-access-6gf4c\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:04.925472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.925437    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q" event={"ID":"f912534e-9398-4d23-a85c-446a048e680e","Type":"ContainerDied","Data":"2b631b3eb7165c7bf2c3030e1d2b2e36e81a37a951fdf66f585e164000523182"}
Oct 31 01:43:04.925641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.925493    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b631b3eb7165c7bf2c3030e1d2b2e36e81a37a951fdf66f585e164000523182"
Oct 31 01:43:04.925641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:04.925541    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"
Oct 31 01:43:11.773769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:11.773731    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:43:12.912082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:12.912037    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" podUID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.189:9004: connect: connection refused"
Oct 31 01:43:13.919231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:13.919181    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:43:14.211428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.211351    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"]
Oct 31 01:43:14.211604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.211574    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m" podUID="246ffb7e-71dc-4006-a855-d00d18dd0152" containerName="payload-processing" containerID="cri-o://f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87" gracePeriod=30
Oct 31 01:43:14.268741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.268706    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"]
Oct 31 01:43:14.268942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.268918    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz" podUID="291fb033-6e9a-458f-8d8c-7c4dc6293e51" containerName="payload-pre-processing" containerID="cri-o://e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d" gracePeriod=30
Oct 31 01:43:14.574290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.574263    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:43:14.577671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.577645    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:43:14.584242 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:43:14.584214    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:43:14.603803 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.603779    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b\" (UniqueName: \"kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b\") pod \"246ffb7e-71dc-4006-a855-d00d18dd0152\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") "
Oct 31 01:43:14.603920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.603829    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume\") pod \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") "
Oct 31 01:43:14.604252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.604227    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume" pod "291fb033-6e9a-458f-8d8c-7c4dc6293e51" (UID: "291fb033-6e9a-458f-8d8c-7c4dc6293e51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:14.605945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.605915    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b" pod "246ffb7e-71dc-4006-a855-d00d18dd0152" (UID: "246ffb7e-71dc-4006-a855-d00d18dd0152"). InnerVolumeSpecName "kube-api-access-9b97b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:14.704887 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.704851    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume\") pod \"246ffb7e-71dc-4006-a855-d00d18dd0152\" (UID: \"246ffb7e-71dc-4006-a855-d00d18dd0152\") "
Oct 31 01:43:14.705089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.704933    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz\" (UniqueName: \"kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz\") pod \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\" (UID: \"291fb033-6e9a-458f-8d8c-7c4dc6293e51\") "
Oct 31 01:43:14.705346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.705305    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume" pod "246ffb7e-71dc-4006-a855-d00d18dd0152" (UID: "246ffb7e-71dc-4006-a855-d00d18dd0152"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:14.705647 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.705621    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b97b\" (UniqueName: \"kubernetes.io/projected/246ffb7e-71dc-4006-a855-d00d18dd0152-kube-api-access-9b97b\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.705790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.705651    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291fb033-6e9a-458f-8d8c-7c4dc6293e51-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.705790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.705668    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246ffb7e-71dc-4006-a855-d00d18dd0152-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:14.708778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.708748    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz" pod "291fb033-6e9a-458f-8d8c-7c4dc6293e51" (UID: "291fb033-6e9a-458f-8d8c-7c4dc6293e51"). InnerVolumeSpecName "kube-api-access-5bppz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:14.811257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:14.811082    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bppz\" (UniqueName: \"kubernetes.io/projected/291fb033-6e9a-458f-8d8c-7c4dc6293e51-kube-api-access-5bppz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.028742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.028702    5419 generic.go:356] "Generic (PLEG): container finished" podID="246ffb7e-71dc-4006-a855-d00d18dd0152" containerID="f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87" exitCode=0
Oct 31 01:43:15.029205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.028780    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"
Oct 31 01:43:15.029205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.028781    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m" event={"ID":"246ffb7e-71dc-4006-a855-d00d18dd0152","Type":"ContainerDied","Data":"f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87"}
Oct 31 01:43:15.029205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.028872    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m" event={"ID":"246ffb7e-71dc-4006-a855-d00d18dd0152","Type":"ContainerDied","Data":"d18438e807a39d4bf18f4b199ee86bcf284bc4b8be404300d4a7f7d203b7d77f"}
Oct 31 01:43:15.029205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.028902    5419 scope.go:122] "RemoveContainer" containerID="f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87"
Oct 31 01:43:15.030510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.030483    5419 generic.go:356] "Generic (PLEG): container finished" podID="291fb033-6e9a-458f-8d8c-7c4dc6293e51" containerID="e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d" exitCode=0
Oct 31 01:43:15.030639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.030535    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz" event={"ID":"291fb033-6e9a-458f-8d8c-7c4dc6293e51","Type":"ContainerDied","Data":"e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d"}
Oct 31 01:43:15.030639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.030578    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz" event={"ID":"291fb033-6e9a-458f-8d8c-7c4dc6293e51","Type":"ContainerDied","Data":"316861e34c71c0f53427439859847c564e201b1f899bfd94e44c66f86922aa7c"}
Oct 31 01:43:15.030639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.030542    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"
Oct 31 01:43:15.039095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.039080    5419 scope.go:122] "RemoveContainer" containerID="f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87"
Oct 31 01:43:15.039391 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:43:15.039372    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87\": container with ID starting with f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87 not found: ID does not exist" containerID="f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87"
Oct 31 01:43:15.039488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.039398    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87"} err="failed to get container status \"f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87\": rpc error: code = NotFound desc = could not find container \"f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87\": container with ID starting with f6f70b3200e3ce4a753c9b9c74d4558478f726fe978728b1f6d22e4571b03b87 not found: ID does not exist"
Oct 31 01:43:15.039488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.039415    5419 scope.go:122] "RemoveContainer" containerID="e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d"
Oct 31 01:43:15.048317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.048301    5419 scope.go:122] "RemoveContainer" containerID="e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d"
Oct 31 01:43:15.048549 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:43:15.048522    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d\": container with ID starting with e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d not found: ID does not exist" containerID="e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d"
Oct 31 01:43:15.048599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.048548    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d"} err="failed to get container status \"e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d\": rpc error: code = NotFound desc = could not find container \"e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d\": container with ID starting with e51bef9f3cb872e83a152fbd03e87dfd0ff5b6a0ccf4141ee9bc67a001626f3d not found: ID does not exist"
Oct 31 01:43:15.199190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.199062    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"]
Oct 31 01:43:15.201641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.201617    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d3023-f7875fb5b-hck6m"]
Oct 31 01:43:15.306368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.306331    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"]
Oct 31 01:43:15.310178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.310152    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d3023-699676b967-fldkz"]
Oct 31 01:43:15.410610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.410575    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"]
Oct 31 01:43:15.410826 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.410800    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv" podUID="c17f2d3b-c29e-4303-9f50-5feea583bf69" containerName="payload-processing" containerID="cri-o://6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f" gracePeriod=30
Oct 31 01:43:15.469044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.469021    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"]
Oct 31 01:43:15.469295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.469272    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft" podUID="1a860479-c2cd-4a00-931d-0be8a188d7c1" containerName="payload-pre-processing" containerID="cri-o://d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3" gracePeriod=30
Oct 31 01:43:15.735287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.735222    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:43:15.755198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.755173    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:43:15.824020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.823989    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4\" (UniqueName: \"kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4\") pod \"1a860479-c2cd-4a00-931d-0be8a188d7c1\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") "
Oct 31 01:43:15.824225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824028    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume\") pod \"c17f2d3b-c29e-4303-9f50-5feea583bf69\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") "
Oct 31 01:43:15.824225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824084    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n\" (UniqueName: \"kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n\") pod \"c17f2d3b-c29e-4303-9f50-5feea583bf69\" (UID: \"c17f2d3b-c29e-4303-9f50-5feea583bf69\") "
Oct 31 01:43:15.824380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824250    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume\") pod \"1a860479-c2cd-4a00-931d-0be8a188d7c1\" (UID: \"1a860479-c2cd-4a00-931d-0be8a188d7c1\") "
Oct 31 01:43:15.824565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824540    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume" pod "c17f2d3b-c29e-4303-9f50-5feea583bf69" (UID: "c17f2d3b-c29e-4303-9f50-5feea583bf69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:15.824644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824613    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume" pod "1a860479-c2cd-4a00-931d-0be8a188d7c1" (UID: "1a860479-c2cd-4a00-931d-0be8a188d7c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:15.824796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824775    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a860479-c2cd-4a00-931d-0be8a188d7c1-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.824796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.824793    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17f2d3b-c29e-4303-9f50-5feea583bf69-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.826364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.826344    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n" pod "c17f2d3b-c29e-4303-9f50-5feea583bf69" (UID: "c17f2d3b-c29e-4303-9f50-5feea583bf69"). InnerVolumeSpecName "kube-api-access-4fl5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:15.826499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.826482    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4" pod "1a860479-c2cd-4a00-931d-0be8a188d7c1" (UID: "1a860479-c2cd-4a00-931d-0be8a188d7c1"). InnerVolumeSpecName "kube-api-access-5z6z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:15.962163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.931611    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fl5n\" (UniqueName: \"kubernetes.io/projected/c17f2d3b-c29e-4303-9f50-5feea583bf69-kube-api-access-4fl5n\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.962163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.931654    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6z4\" (UniqueName: \"kubernetes.io/projected/1a860479-c2cd-4a00-931d-0be8a188d7c1-kube-api-access-5z6z4\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:15.964436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.964350    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"]
Oct 31 01:43:15.981943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:15.975545    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3d3023-j8n4q"]
Oct 31 01:43:16.045051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.044953    5419 generic.go:356] "Generic (PLEG): container finished" podID="c17f2d3b-c29e-4303-9f50-5feea583bf69" containerID="6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f" exitCode=0
Oct 31 01:43:16.045051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.045009    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"
Oct 31 01:43:16.045051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.045009    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv" event={"ID":"c17f2d3b-c29e-4303-9f50-5feea583bf69","Type":"ContainerDied","Data":"6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f"}
Oct 31 01:43:16.045692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.045073    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv" event={"ID":"c17f2d3b-c29e-4303-9f50-5feea583bf69","Type":"ContainerDied","Data":"4d68fae4d7b49788241f63f8174a1d5f3b2e4d7849894a4ac3dea5ab4d91feed"}
Oct 31 01:43:16.045692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.045101    5419 scope.go:122] "RemoveContainer" containerID="6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f"
Oct 31 01:43:16.048260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.048236    5419 generic.go:356] "Generic (PLEG): container finished" podID="1a860479-c2cd-4a00-931d-0be8a188d7c1" containerID="d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3" exitCode=0
Oct 31 01:43:16.048505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.048265    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft" event={"ID":"1a860479-c2cd-4a00-931d-0be8a188d7c1","Type":"ContainerDied","Data":"d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3"}
Oct 31 01:43:16.048631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.048541    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft" event={"ID":"1a860479-c2cd-4a00-931d-0be8a188d7c1","Type":"ContainerDied","Data":"0491bd2a1f5828d75aa659b87043265123054ddf0116a9da4dac4f4eb20f1a08"}
Oct 31 01:43:16.048631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.048296    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"
Oct 31 01:43:16.059411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.059382    5419 scope.go:122] "RemoveContainer" containerID="6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f"
Oct 31 01:43:16.059764 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:43:16.059742    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f\": container with ID starting with 6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f not found: ID does not exist" containerID="6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f"
Oct 31 01:43:16.059870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.059776    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f"} err="failed to get container status \"6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f\": rpc error: code = NotFound desc = could not find container \"6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f\": container with ID starting with 6a1c8f37d73f9fef8af50e1954c40e8a238de23e7a907310d0910156016f876f not found: ID does not exist"
Oct 31 01:43:16.059870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.059800    5419 scope.go:122] "RemoveContainer" containerID="d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3"
Oct 31 01:43:16.070418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.070397    5419 scope.go:122] "RemoveContainer" containerID="d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3"
Oct 31 01:43:16.070716 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:43:16.070697    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3\": container with ID starting with d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3 not found: ID does not exist" containerID="d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3"
Oct 31 01:43:16.070787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.070724    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3"} err="failed to get container status \"d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3\": rpc error: code = NotFound desc = could not find container \"d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3\": container with ID starting with d1c82e154cb850e722fc0cb14a4fbaa47c7dbaca789ffd691b6b2452220ac8e3 not found: ID does not exist"
Oct 31 01:43:16.219036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.218996    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"]
Oct 31 01:43:16.223715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.223683    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9e5368-9b9c69d6-t2ksv"]
Oct 31 01:43:16.330442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.330370    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"]
Oct 31 01:43:16.333446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.333417    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9e5368-84b68d47b7-dffft"]
Oct 31 01:43:16.837330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.837300    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a860479-c2cd-4a00-931d-0be8a188d7c1" path="/var/lib/kubelet/pods/1a860479-c2cd-4a00-931d-0be8a188d7c1/volumes"
Oct 31 01:43:16.837725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.837712    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246ffb7e-71dc-4006-a855-d00d18dd0152" path="/var/lib/kubelet/pods/246ffb7e-71dc-4006-a855-d00d18dd0152/volumes"
Oct 31 01:43:16.838080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.838068    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291fb033-6e9a-458f-8d8c-7c4dc6293e51" path="/var/lib/kubelet/pods/291fb033-6e9a-458f-8d8c-7c4dc6293e51/volumes"
Oct 31 01:43:16.838457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.838445    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17f2d3b-c29e-4303-9f50-5feea583bf69" path="/var/lib/kubelet/pods/c17f2d3b-c29e-4303-9f50-5feea583bf69/volumes"
Oct 31 01:43:16.839263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:16.839249    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f912534e-9398-4d23-a85c-446a048e680e" path="/var/lib/kubelet/pods/f912534e-9398-4d23-a85c-446a048e680e/volumes"
Oct 31 01:43:17.313273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:17.313231    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"]
Oct 31 01:43:17.315474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:17.315451    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9e5368-fllfz"]
Oct 31 01:43:18.839007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:18.838975    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38672beb-8e1f-471c-ac15-ac4d7297b103" path="/var/lib/kubelet/pods/38672beb-8e1f-471c-ac15-ac4d7297b103/volumes"
Oct 31 01:43:22.911308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:22.911278    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:43:23.011404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.011374    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"]
Oct 31 01:43:23.011632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.011609    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" containerID="cri-o://ef1991dbd485668de006b8f549cf3cfe9558c91cfebe837582a83d7313468672" gracePeriod=30
Oct 31 01:43:23.952654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.952622    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"]
Oct 31 01:43:23.953209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953193    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1a860479-c2cd-4a00-931d-0be8a188d7c1" containerName="payload-pre-processing"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953210    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a860479-c2cd-4a00-931d-0be8a188d7c1" containerName="payload-pre-processing"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953227    5419 cpu_manager.go:414] "containerMap: removing container" podUID="246ffb7e-71dc-4006-a855-d00d18dd0152" containerName="payload-processing"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953232    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246ffb7e-71dc-4006-a855-d00d18dd0152" containerName="payload-processing"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953250    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c17f2d3b-c29e-4303-9f50-5feea583bf69" containerName="payload-processing"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953255    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17f2d3b-c29e-4303-9f50-5feea583bf69" containerName="payload-processing"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953276    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f912534e-9398-4d23-a85c-446a048e680e" containerName="revoke-keys"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953282    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f912534e-9398-4d23-a85c-446a048e680e" containerName="revoke-keys"
Oct 31 01:43:23.953291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953293    5419 cpu_manager.go:414] "containerMap: removing container" podUID="291fb033-6e9a-458f-8d8c-7c4dc6293e51" containerName="payload-pre-processing"
Oct 31 01:43:23.953616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953299    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291fb033-6e9a-458f-8d8c-7c4dc6293e51" containerName="payload-pre-processing"
Oct 31 01:43:23.953616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953371    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17f2d3b-c29e-4303-9f50-5feea583bf69" containerName="payload-processing"
Oct 31 01:43:23.953616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953383    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f912534e-9398-4d23-a85c-446a048e680e" containerName="revoke-keys"
Oct 31 01:43:23.953616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953392    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="291fb033-6e9a-458f-8d8c-7c4dc6293e51" containerName="payload-pre-processing"
Oct 31 01:43:23.953616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953399    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a860479-c2cd-4a00-931d-0be8a188d7c1" containerName="payload-pre-processing"
Oct 31 01:43:23.953616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.953406    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="246ffb7e-71dc-4006-a855-d00d18dd0152" containerName="payload-processing"
Oct 31 01:43:23.957995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.957980    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:23.996332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.996299    5419 patch_prober.go:29] interesting pod/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:43:23.996482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:23.996351    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:43:24.049716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.049676    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.049890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.049824    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.049938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.049923    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbj46\" (UniqueName: \"kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.063400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.063371    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:24.063400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.063385    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:24.063552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.063414    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:24.063597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.063556    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:43:24.066162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.066128    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"]
Oct 31 01:43:24.151032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.151001    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.151263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.151078    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbj46\" (UniqueName: \"kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.151263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.151180    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.151782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.151762    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.153675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.153657    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.212111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.212042    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbj46\" (UniqueName: \"kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46\") pod \"maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.271763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.271736    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:24.647638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:24.647612    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"]
Oct 31 01:43:24.649363 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:24.649328    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0215dc92_bc7e_4100_ad14_9e66ae24d52a.slice/crio-5774ae2447e2a2449c9fb8954848e702f151fd816f5daf5e34052a4b5f30f6b8 WatchSource:0}: Error finding container 5774ae2447e2a2449c9fb8954848e702f151fd816f5daf5e34052a4b5f30f6b8: Status 404 returned error can't find the container with id 5774ae2447e2a2449c9fb8954848e702f151fd816f5daf5e34052a4b5f30f6b8
Oct 31 01:43:25.160844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:25.160806    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc" event={"ID":"0215dc92-bc7e-4100-ad14-9e66ae24d52a","Type":"ContainerStarted","Data":"c05e7e3bf658d0cf88180995ff05b55ce1fa6d88702b4b1d3153cc152a153eaf"}
Oct 31 01:43:25.161336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:25.160874    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc" event={"ID":"0215dc92-bc7e-4100-ad14-9e66ae24d52a","Type":"ContainerStarted","Data":"5774ae2447e2a2449c9fb8954848e702f151fd816f5daf5e34052a4b5f30f6b8"}
Oct 31 01:43:25.278098 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:25.277986    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc" podStartSLOduration=2.277968233 podStartE2EDuration="2.277968233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:25.277669447 +0000 UTC m=+2326.578626026" watchObservedRunningTime="2026-10-31 01:43:25.277968233 +0000 UTC m=+2326.578924811"
Oct 31 01:43:26.168289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:26.168252    5419 generic.go:356] "Generic (PLEG): container finished" podID="0215dc92-bc7e-4100-ad14-9e66ae24d52a" containerID="c05e7e3bf658d0cf88180995ff05b55ce1fa6d88702b4b1d3153cc152a153eaf" exitCode=0
Oct 31 01:43:26.168717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:26.168342    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc" event={"ID":"0215dc92-bc7e-4100-ad14-9e66ae24d52a","Type":"ContainerDied","Data":"c05e7e3bf658d0cf88180995ff05b55ce1fa6d88702b4b1d3153cc152a153eaf"}
Oct 31 01:43:27.325758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.325734    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:27.384564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.384532    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca\") pod \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") "
Oct 31 01:43:27.384745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.384732    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46\" (UniqueName: \"kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46\") pod \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") "
Oct 31 01:43:27.384798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.384755    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca\") pod \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\" (UID: \"0215dc92-bc7e-4100-ad14-9e66ae24d52a\") "
Oct 31 01:43:27.385217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.385194    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca" pod "0215dc92-bc7e-4100-ad14-9e66ae24d52a" (UID: "0215dc92-bc7e-4100-ad14-9e66ae24d52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:27.386759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.386742    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca" pod "0215dc92-bc7e-4100-ad14-9e66ae24d52a" (UID: "0215dc92-bc7e-4100-ad14-9e66ae24d52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:27.386852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.386826    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46" pod "0215dc92-bc7e-4100-ad14-9e66ae24d52a" (UID: "0215dc92-bc7e-4100-ad14-9e66ae24d52a"). InnerVolumeSpecName "kube-api-access-bbj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:27.486095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.486062    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:27.486095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.486089    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbj46\" (UniqueName: \"kubernetes.io/projected/0215dc92-bc7e-4100-ad14-9e66ae24d52a-kube-api-access-bbj46\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:27.486095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:27.486099    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0215dc92-bc7e-4100-ad14-9e66ae24d52a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.183985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.183727    5419 generic.go:356] "Generic (PLEG): container finished" podID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerID="ef1991dbd485668de006b8f549cf3cfe9558c91cfebe837582a83d7313468672" exitCode=0
Oct 31 01:43:28.183985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.183819    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" event={"ID":"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a","Type":"ContainerDied","Data":"ef1991dbd485668de006b8f549cf3cfe9558c91cfebe837582a83d7313468672"}
Oct 31 01:43:28.183985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.183879    5419 scope.go:122] "RemoveContainer" containerID="ee47b1509e454c68964592b36e2a114af6f878519b0be58871c4fdad09695435"
Oct 31 01:43:28.187457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.187426    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc" event={"ID":"0215dc92-bc7e-4100-ad14-9e66ae24d52a","Type":"ContainerDied","Data":"5774ae2447e2a2449c9fb8954848e702f151fd816f5daf5e34052a4b5f30f6b8"}
Oct 31 01:43:28.187566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.187486    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5774ae2447e2a2449c9fb8954848e702f151fd816f5daf5e34052a4b5f30f6b8"
Oct 31 01:43:28.187566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.187556    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"
Oct 31 01:43:28.252894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.252868    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:43:28.398859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.398818    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.398929    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.398973    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399042    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399100    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399132    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399260    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399309    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399354    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy\") pod \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\" (UID: \"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a\") "
Oct 31 01:43:28.399830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399573    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:28.399948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399918    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:28.400029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.399936    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.400166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.400118    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:28.400166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.400031    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:28.400551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.400526    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:28.402332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.402297    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:43:28.402524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.402503    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:28.402922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.402898    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "kube-api-access-sfld8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:28.402922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.402912    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy" pod "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" (UID: "1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:43:28.501006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.500967    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501000    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501014    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfld8\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-kube-api-access-sfld8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501029    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501040    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501052    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501063    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:28.501258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:28.501074    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:29.195256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:29.195183    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"
Oct 31 01:43:29.195412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:29.195181    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw" event={"ID":"1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a","Type":"ContainerDied","Data":"7df0d12ac52504ec05a18b45e5e02b4d9cf138f0a02aed53ea194217f75ebfd9"}
Oct 31 01:43:29.195412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:29.195305    5419 scope.go:122] "RemoveContainer" containerID="ef1991dbd485668de006b8f549cf3cfe9558c91cfebe837582a83d7313468672"
Oct 31 01:43:29.362270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:29.362236    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"]
Oct 31 01:43:29.411662 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:29.411626    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3d3023-openshift-default-6867c59cbf-nvmxw"]
Oct 31 01:43:30.837816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:30.837783    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" path="/var/lib/kubelet/pods/1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a/volumes"
Oct 31 01:43:34.973100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973068    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"]
Oct 31 01:43:34.973608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973594    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy"
Oct 31 01:43:34.973666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973609    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy"
Oct 31 01:43:34.973666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973626    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy"
Oct 31 01:43:34.973666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973632    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy"
Oct 31 01:43:34.973666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973667    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0215dc92-bc7e-4100-ad14-9e66ae24d52a" containerName="revoke-keys"
Oct 31 01:43:34.973805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973672    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0215dc92-bc7e-4100-ad14-9e66ae24d52a" containerName="revoke-keys"
Oct 31 01:43:34.973805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973756    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0215dc92-bc7e-4100-ad14-9e66ae24d52a" containerName="revoke-keys"
Oct 31 01:43:34.973805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973765    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy"
Oct 31 01:43:34.973805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.973772    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d59487d-4f4d-40ca-98e3-37b0c7fcfc8a" containerName="istio-proxy"
Oct 31 01:43:34.977058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:34.977035    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.032784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.032753    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:35.080125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.080089    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.080315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.080178    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.080315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.080259    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fp6b\" (UniqueName: \"kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.082442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.082404    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:35.082557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.082457    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:35.084981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.084957    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"]
Oct 31 01:43:35.131932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.131903    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:43:35.181723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.181688    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.181889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.181737    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.181889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.181797    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fp6b\" (UniqueName: \"kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.182486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.182464    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.184328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.184311    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.238341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.238278    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fp6b\" (UniqueName: \"kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b\") pod \"maas-api-revoke-sub-e6141e20-kntqw\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.292386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.292352    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:35.670272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:35.670246    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"]
Oct 31 01:43:35.672558 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:35.672525    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682742b0_56a9_4fdd_b341_f83b60f41c75.slice/crio-9d0b6259cb845419d9b0801edb2c9500e5a0278457bf984d8e3052efdb8e17f0 WatchSource:0}: Error finding container 9d0b6259cb845419d9b0801edb2c9500e5a0278457bf984d8e3052efdb8e17f0: Status 404 returned error can't find the container with id 9d0b6259cb845419d9b0801edb2c9500e5a0278457bf984d8e3052efdb8e17f0
Oct 31 01:43:36.242484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:36.242439    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw" event={"ID":"682742b0-56a9-4fdd-b341-f83b60f41c75","Type":"ContainerStarted","Data":"1e8bf4bb2a9f7593cda572fec5e0494ac37be3d59a4827ce9e812197c61e42a0"}
Oct 31 01:43:36.242943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:36.242528    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw" event={"ID":"682742b0-56a9-4fdd-b341-f83b60f41c75","Type":"ContainerStarted","Data":"9d0b6259cb845419d9b0801edb2c9500e5a0278457bf984d8e3052efdb8e17f0"}
Oct 31 01:43:36.357209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:36.357075    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw" podStartSLOduration=2.357059544 podStartE2EDuration="2.357059544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:36.3560057 +0000 UTC m=+2337.656962300" watchObservedRunningTime="2026-10-31 01:43:36.357059544 +0000 UTC m=+2337.658016162"
Oct 31 01:43:37.249836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:37.249800    5419 generic.go:356] "Generic (PLEG): container finished" podID="682742b0-56a9-4fdd-b341-f83b60f41c75" containerID="1e8bf4bb2a9f7593cda572fec5e0494ac37be3d59a4827ce9e812197c61e42a0" exitCode=0
Oct 31 01:43:37.250328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:37.249858    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw" event={"ID":"682742b0-56a9-4fdd-b341-f83b60f41c75","Type":"ContainerDied","Data":"1e8bf4bb2a9f7593cda572fec5e0494ac37be3d59a4827ce9e812197c61e42a0"}
Oct 31 01:43:38.409302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.409269    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:38.522209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.522105    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca\") pod \"682742b0-56a9-4fdd-b341-f83b60f41c75\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") "
Oct 31 01:43:38.522209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.522172    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca\") pod \"682742b0-56a9-4fdd-b341-f83b60f41c75\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") "
Oct 31 01:43:38.522409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.522217    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b\" (UniqueName: \"kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b\") pod \"682742b0-56a9-4fdd-b341-f83b60f41c75\" (UID: \"682742b0-56a9-4fdd-b341-f83b60f41c75\") "
Oct 31 01:43:38.522465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.522444    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca" pod "682742b0-56a9-4fdd-b341-f83b60f41c75" (UID: "682742b0-56a9-4fdd-b341-f83b60f41c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:38.522676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.522655    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:38.524394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.524371    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca" pod "682742b0-56a9-4fdd-b341-f83b60f41c75" (UID: "682742b0-56a9-4fdd-b341-f83b60f41c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:38.524483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.524470    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b" pod "682742b0-56a9-4fdd-b341-f83b60f41c75" (UID: "682742b0-56a9-4fdd-b341-f83b60f41c75"). InnerVolumeSpecName "kube-api-access-9fp6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:38.624356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.624325    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682742b0-56a9-4fdd-b341-f83b60f41c75-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:38.624356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:38.624351    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fp6b\" (UniqueName: \"kubernetes.io/projected/682742b0-56a9-4fdd-b341-f83b60f41c75-kube-api-access-9fp6b\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:39.263410 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:39.263381    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"
Oct 31 01:43:39.263614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:39.263366    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw" event={"ID":"682742b0-56a9-4fdd-b341-f83b60f41c75","Type":"ContainerDied","Data":"9d0b6259cb845419d9b0801edb2c9500e5a0278457bf984d8e3052efdb8e17f0"}
Oct 31 01:43:39.263614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:39.263507    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d0b6259cb845419d9b0801edb2c9500e5a0278457bf984d8e3052efdb8e17f0"
Oct 31 01:43:40.478627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:40.478590    5419 scope.go:122] "RemoveContainer" containerID="cabeb97ff65fb4df92094304d032060d725e1ae3c433f2bc7b87b61d94b651b3"
Oct 31 01:43:44.582058 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:43:44.582019    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:43:45.725533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.725492    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"]
Oct 31 01:43:45.726038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.726024    5419 cpu_manager.go:414] "containerMap: removing container" podUID="682742b0-56a9-4fdd-b341-f83b60f41c75" containerName="revoke-keys"
Oct 31 01:43:45.726084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.726038    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682742b0-56a9-4fdd-b341-f83b60f41c75" containerName="revoke-keys"
Oct 31 01:43:45.726163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.726154    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="682742b0-56a9-4fdd-b341-f83b60f41c75" containerName="revoke-keys"
Oct 31 01:43:45.731099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.731084    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.783257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.783227    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:45.783856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.783289    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:45.783856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.783301    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:45.786881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.786857    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"]
Oct 31 01:43:45.804019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.803990    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbtm\" (UniqueName: \"kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.804130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.804025    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.804130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.804046    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.883558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.883530    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:43:45.905232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.905204    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbtm\" (UniqueName: \"kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.905405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.905247    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.905405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.905272    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.905944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.905924    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.907875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.907854    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:45.963814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:45.963792    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbtm\" (UniqueName: \"kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm\") pod \"maas-api-revoke-sub-b9198562-nnl6p\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:46.044395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:46.044327    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:46.420412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:46.420386    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"]
Oct 31 01:43:46.422790 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:46.422752    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f956485_7ab2_4987_81af_4723748e67fb.slice/crio-c924afdb6c15250a1d4e46698bf50ef6fbb3190bf8f07c8db5baf12e0817fbfa WatchSource:0}: Error finding container c924afdb6c15250a1d4e46698bf50ef6fbb3190bf8f07c8db5baf12e0817fbfa: Status 404 returned error can't find the container with id c924afdb6c15250a1d4e46698bf50ef6fbb3190bf8f07c8db5baf12e0817fbfa
Oct 31 01:43:47.320223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:47.320186    5419 generic.go:356] "Generic (PLEG): container finished" podID="1f956485-7ab2-4987-81af-4723748e67fb" containerID="cbe12d1e1c5462d575b6989e364ee49f0e410d643e55fba31c4779b687af6487" exitCode=0
Oct 31 01:43:47.320629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:47.320271    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p" event={"ID":"1f956485-7ab2-4987-81af-4723748e67fb","Type":"ContainerDied","Data":"cbe12d1e1c5462d575b6989e364ee49f0e410d643e55fba31c4779b687af6487"}
Oct 31 01:43:47.320629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:47.320320    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p" event={"ID":"1f956485-7ab2-4987-81af-4723748e67fb","Type":"ContainerStarted","Data":"c924afdb6c15250a1d4e46698bf50ef6fbb3190bf8f07c8db5baf12e0817fbfa"}
Oct 31 01:43:47.714321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:47.714241    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:43:48.477231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.477210    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:48.537868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.537839    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca\") pod \"1f956485-7ab2-4987-81af-4723748e67fb\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") "
Oct 31 01:43:48.538021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.537944    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca\") pod \"1f956485-7ab2-4987-81af-4723748e67fb\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") "
Oct 31 01:43:48.538099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.538036    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm\" (UniqueName: \"kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm\") pod \"1f956485-7ab2-4987-81af-4723748e67fb\" (UID: \"1f956485-7ab2-4987-81af-4723748e67fb\") "
Oct 31 01:43:48.538479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.538455    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca" pod "1f956485-7ab2-4987-81af-4723748e67fb" (UID: "1f956485-7ab2-4987-81af-4723748e67fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:48.540357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.540332    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca" pod "1f956485-7ab2-4987-81af-4723748e67fb" (UID: "1f956485-7ab2-4987-81af-4723748e67fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:48.540468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.540390    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm" pod "1f956485-7ab2-4987-81af-4723748e67fb" (UID: "1f956485-7ab2-4987-81af-4723748e67fb"). InnerVolumeSpecName "kube-api-access-6kbtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:48.639550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.639453    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:48.639550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.639497    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbtm\" (UniqueName: \"kubernetes.io/projected/1f956485-7ab2-4987-81af-4723748e67fb-kube-api-access-6kbtm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:48.639550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:48.639516    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f956485-7ab2-4987-81af-4723748e67fb-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:49.334676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:49.334645    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"
Oct 31 01:43:49.334676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:49.334657    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p" event={"ID":"1f956485-7ab2-4987-81af-4723748e67fb","Type":"ContainerDied","Data":"c924afdb6c15250a1d4e46698bf50ef6fbb3190bf8f07c8db5baf12e0817fbfa"}
Oct 31 01:43:49.334899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:49.334700    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c924afdb6c15250a1d4e46698bf50ef6fbb3190bf8f07c8db5baf12e0817fbfa"
Oct 31 01:43:50.350041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.350007    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"]
Oct 31 01:43:50.350569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.350554    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1f956485-7ab2-4987-81af-4723748e67fb" containerName="revoke-keys"
Oct 31 01:43:50.350618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.350570    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f956485-7ab2-4987-81af-4723748e67fb" containerName="revoke-keys"
Oct 31 01:43:50.350698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.350689    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f956485-7ab2-4987-81af-4723748e67fb" containerName="revoke-keys"
Oct 31 01:43:50.354104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.354088    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.407388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.407331    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:43:50.456772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.456743    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:43:50.456958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.456791    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:43:50.456958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.456813    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:43:50.459463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.459437    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.459627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.459610    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.459733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.459713    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4nk\" (UniqueName: \"kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.459785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.459762    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"]
Oct 31 01:43:50.560476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.560445    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.560681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.560530    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4nk\" (UniqueName: \"kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.560681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.560598    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.561340 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.561301    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.563423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.563397    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.620018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.619940    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4nk\" (UniqueName: \"kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk\") pod \"maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:50.667997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:50.667960    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:51.043770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:51.043742    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"]
Oct 31 01:43:51.045828 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:43:51.045803    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b6414e_d3e2_4526_8f35_1d497320f926.slice/crio-48f3978ab6a301575dcbbc405b7cb007ff32621711cab4c6f566bfba00fda25f WatchSource:0}: Error finding container 48f3978ab6a301575dcbbc405b7cb007ff32621711cab4c6f566bfba00fda25f: Status 404 returned error can't find the container with id 48f3978ab6a301575dcbbc405b7cb007ff32621711cab4c6f566bfba00fda25f
Oct 31 01:43:51.348029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:51.347997    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4" event={"ID":"f7b6414e-d3e2-4526-8f35-1d497320f926","Type":"ContainerStarted","Data":"48f3978ab6a301575dcbbc405b7cb007ff32621711cab4c6f566bfba00fda25f"}
Oct 31 01:43:52.359071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:52.358980    5419 generic.go:356] "Generic (PLEG): container finished" podID="f7b6414e-d3e2-4526-8f35-1d497320f926" containerID="6f93c1fbbe00ce230ea42423dffe697e159442c0fe1e33ace2bb59bd6eb9677b" exitCode=0
Oct 31 01:43:52.359071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:52.359059    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4" event={"ID":"f7b6414e-d3e2-4526-8f35-1d497320f926","Type":"ContainerDied","Data":"6f93c1fbbe00ce230ea42423dffe697e159442c0fe1e33ace2bb59bd6eb9677b"}
Oct 31 01:43:53.526074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.526050    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:53.593286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.593258    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca\") pod \"f7b6414e-d3e2-4526-8f35-1d497320f926\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") "
Oct 31 01:43:53.593418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.593324    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk\" (UniqueName: \"kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk\") pod \"f7b6414e-d3e2-4526-8f35-1d497320f926\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") "
Oct 31 01:43:53.593499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.593433    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca\") pod \"f7b6414e-d3e2-4526-8f35-1d497320f926\" (UID: \"f7b6414e-d3e2-4526-8f35-1d497320f926\") "
Oct 31 01:43:53.593673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.593642    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca" pod "f7b6414e-d3e2-4526-8f35-1d497320f926" (UID: "f7b6414e-d3e2-4526-8f35-1d497320f926"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:43:53.593919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.593900    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:53.595497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.595478    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca" pod "f7b6414e-d3e2-4526-8f35-1d497320f926" (UID: "f7b6414e-d3e2-4526-8f35-1d497320f926"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:43:53.595599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.595582    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk" pod "f7b6414e-d3e2-4526-8f35-1d497320f926" (UID: "f7b6414e-d3e2-4526-8f35-1d497320f926"). InnerVolumeSpecName "kube-api-access-df4nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:43:53.695480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.695409    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df4nk\" (UniqueName: \"kubernetes.io/projected/f7b6414e-d3e2-4526-8f35-1d497320f926-kube-api-access-df4nk\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:53.695480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:53.695437    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b6414e-d3e2-4526-8f35-1d497320f926-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:43:54.372610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:54.372575    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4" event={"ID":"f7b6414e-d3e2-4526-8f35-1d497320f926","Type":"ContainerDied","Data":"48f3978ab6a301575dcbbc405b7cb007ff32621711cab4c6f566bfba00fda25f"}
Oct 31 01:43:54.372610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:54.372611    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"
Oct 31 01:43:54.372828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:54.372635    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48f3978ab6a301575dcbbc405b7cb007ff32621711cab4c6f566bfba00fda25f"
Oct 31 01:43:56.407824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:56.407784    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"]
Oct 31 01:43:56.457227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:56.457192    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-df5d0b-j55cc"]
Oct 31 01:43:56.837645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:56.837617    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0215dc92-bc7e-4100-ad14-9e66ae24d52a" path="/var/lib/kubelet/pods/0215dc92-bc7e-4100-ad14-9e66ae24d52a/volumes"
Oct 31 01:43:58.744684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:58.744645    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"]
Oct 31 01:43:59.912536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:43:59.912498    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"]
Oct 31 01:44:02.949273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:02.949244    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"]
Oct 31 01:44:02.949646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:02.949452    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy" containerID="cri-o://8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d" gracePeriod=30
Oct 31 01:44:03.746621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:03.746579    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8" podUID="a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" containerName="payload-pre-processing" containerID="cri-o://028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6" gracePeriod=25
Oct 31 01:44:04.020325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.020302    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:44:04.152554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.152523    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf\" (UniqueName: \"kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf\") pod \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") "
Oct 31 01:44:04.152770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.152750    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume\") pod \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\" (UID: \"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409\") "
Oct 31 01:44:04.156202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.156170    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume" pod "a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" (UID: "a1ae76c2-4f9c-4e77-a48b-99ece7c0e409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:04.158458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.158424    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf" pod "a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" (UID: "a1ae76c2-4f9c-4e77-a48b-99ece7c0e409"). InnerVolumeSpecName "kube-api-access-8lwsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:04.254424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.254342    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:04.254424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.254372    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lwsf\" (UniqueName: \"kubernetes.io/projected/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409-kube-api-access-8lwsf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:04.458191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.458126    5419 generic.go:356] "Generic (PLEG): container finished" podID="a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" containerID="028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6" exitCode=0
Oct 31 01:44:04.458191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.458170    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8" event={"ID":"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409","Type":"ContainerDied","Data":"028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6"}
Oct 31 01:44:04.458422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.458243    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8" event={"ID":"a1ae76c2-4f9c-4e77-a48b-99ece7c0e409","Type":"ContainerDied","Data":"d2256b33c2beda595bf10cd8bf3794bcd26cbf2b9e7836ded505099c620cf69e"}
Oct 31 01:44:04.458422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.458186    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"
Oct 31 01:44:04.458422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.458274    5419 scope.go:122] "RemoveContainer" containerID="028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6"
Oct 31 01:44:04.469043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.469022    5419 scope.go:122] "RemoveContainer" containerID="028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6"
Oct 31 01:44:04.469331 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:04.469312    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6\": container with ID starting with 028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6 not found: ID does not exist" containerID="028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6"
Oct 31 01:44:04.469409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.469342    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6"} err="failed to get container status \"028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6\": rpc error: code = NotFound desc = could not find container \"028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6\": container with ID starting with 028f81a3e792d9693f366ade2c33f2f12da451f47bce542cf9cc42faf7b4b9f6 not found: ID does not exist"
Oct 31 01:44:04.623319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.623241    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"]
Oct 31 01:44:04.626712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.626688    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b2d49b-8d867f45f-c2lr8"]
Oct 31 01:44:04.837581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:04.837544    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" path="/var/lib/kubelet/pods/a1ae76c2-4f9c-4e77-a48b-99ece7c0e409/volumes"
Oct 31 01:44:06.015386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.015342    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"]
Oct 31 01:44:06.015946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.015595    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq" podUID="21ddc321-5806-498a-9ea4-38788c7d0c17" containerName="payload-processing" containerID="cri-o://6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490" gracePeriod=30
Oct 31 01:44:06.066436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.066400    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"]
Oct 31 01:44:06.066688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.066621    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" podUID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" containerName="payload-pre-processing" containerID="cri-o://e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b" gracePeriod=30
Oct 31 01:44:06.360114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.360089    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:44:06.362938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.362918    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:44:06.388241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.388216    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx\" (UniqueName: \"kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx\") pod \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") "
Oct 31 01:44:06.388369 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.388341    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume\") pod \"21ddc321-5806-498a-9ea4-38788c7d0c17\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") "
Oct 31 01:44:06.388439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.388413    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume\") pod \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\" (UID: \"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e\") "
Oct 31 01:44:06.388490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.388440    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8\" (UniqueName: \"kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8\") pod \"21ddc321-5806-498a-9ea4-38788c7d0c17\" (UID: \"21ddc321-5806-498a-9ea4-38788c7d0c17\") "
Oct 31 01:44:06.388751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.388724    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume" pod "21ddc321-5806-498a-9ea4-38788c7d0c17" (UID: "21ddc321-5806-498a-9ea4-38788c7d0c17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:06.388843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.388759    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume" pod "2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" (UID: "2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:06.390746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.390720    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8" pod "21ddc321-5806-498a-9ea4-38788c7d0c17" (UID: "21ddc321-5806-498a-9ea4-38788c7d0c17"). InnerVolumeSpecName "kube-api-access-tbkp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:06.390882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.390861    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx" pod "2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" (UID: "2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e"). InnerVolumeSpecName "kube-api-access-6fdhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:06.473832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.473802    5419 generic.go:356] "Generic (PLEG): container finished" podID="21ddc321-5806-498a-9ea4-38788c7d0c17" containerID="6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490" exitCode=0
Oct 31 01:44:06.474025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.473858    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"
Oct 31 01:44:06.474025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.473895    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq" event={"ID":"21ddc321-5806-498a-9ea4-38788c7d0c17","Type":"ContainerDied","Data":"6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490"}
Oct 31 01:44:06.474025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.473981    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq" event={"ID":"21ddc321-5806-498a-9ea4-38788c7d0c17","Type":"ContainerDied","Data":"d9fd92d1db8da6d49e126c6042a833d85696b47edc1ab321c254acdee0d7dc31"}
Oct 31 01:44:06.474025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.474009    5419 scope.go:122] "RemoveContainer" containerID="6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490"
Oct 31 01:44:06.475468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.475434    5419 generic.go:356] "Generic (PLEG): container finished" podID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" containerID="e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b" exitCode=0
Oct 31 01:44:06.475772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.475466    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" event={"ID":"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e","Type":"ContainerDied","Data":"e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b"}
Oct 31 01:44:06.475772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.475511    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7" event={"ID":"2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e","Type":"ContainerDied","Data":"b24b42b1a03c59dd3741d87c5153d88a827547e6506b27b34dda802878517826"}
Oct 31 01:44:06.475772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.475485    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"
Oct 31 01:44:06.487307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.487264    5419 scope.go:122] "RemoveContainer" containerID="6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490"
Oct 31 01:44:06.488533 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:06.488442    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490\": container with ID starting with 6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490 not found: ID does not exist" containerID="6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490"
Oct 31 01:44:06.488533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.488488    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490"} err="failed to get container status \"6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490\": rpc error: code = NotFound desc = could not find container \"6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490\": container with ID starting with 6e12127c1d32771d75ed52195eedd6dc99035b679f392fb79c4a1f4cb530b490 not found: ID does not exist"
Oct 31 01:44:06.488533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.488514    5419 scope.go:122] "RemoveContainer" containerID="e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b"
Oct 31 01:44:06.489839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.489706    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21ddc321-5806-498a-9ea4-38788c7d0c17-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:06.489839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.489739    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:06.489839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.489757    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkp8\" (UniqueName: \"kubernetes.io/projected/21ddc321-5806-498a-9ea4-38788c7d0c17-kube-api-access-tbkp8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:06.489839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.489772    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fdhx\" (UniqueName: \"kubernetes.io/projected/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e-kube-api-access-6fdhx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:06.500671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.500657    5419 scope.go:122] "RemoveContainer" containerID="e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b"
Oct 31 01:44:06.500992 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:06.500969    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b\": container with ID starting with e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b not found: ID does not exist" containerID="e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b"
Oct 31 01:44:06.501085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.500998    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b"} err="failed to get container status \"e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b\": rpc error: code = NotFound desc = could not find container \"e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b\": container with ID starting with e7b445081a134b40478e30389a7908e00e76fe8ea22896d51b3d301167dde60b not found: ID does not exist"
Oct 31 01:44:06.756118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.756075    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"]
Oct 31 01:44:06.759894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.759867    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8af859-6d6894c5d-wfdlq"]
Oct 31 01:44:06.864884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.864844    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ddc321-5806-498a-9ea4-38788c7d0c17" path="/var/lib/kubelet/pods/21ddc321-5806-498a-9ea4-38788c7d0c17/volumes"
Oct 31 01:44:06.867208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.867181    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"]
Oct 31 01:44:06.870485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:06.870463    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8af859-746cb6f599-hh6x7"]
Oct 31 01:44:07.731182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:07.731130    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"]
Oct 31 01:44:07.780193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:07.780162    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8af859-j4dvj"]
Oct 31 01:44:08.228491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.228460    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:44:08.312600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312560    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.312812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312621    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.312812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312667    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.312812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312690    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.312812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312732    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.312812 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312812    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.313124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312860    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.313124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312930    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.313124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312957    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token\") pod \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\" (UID: \"e9aa2ebd-853c-45dd-a273-b0aae3d34984\") "
Oct 31 01:44:08.313124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.312963    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:08.313124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313034    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:08.313575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313242    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:08.313575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313256    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:08.313575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313485    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.313575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313508    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.313575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313524    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.313575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313542    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.313910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.313836    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:08.315105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.315075    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "kube-api-access-lw7j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:08.315269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.315245    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:08.315358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.315337    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:08.315459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.315439    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo" pod "e9aa2ebd-853c-45dd-a273-b0aae3d34984" (UID: "e9aa2ebd-853c-45dd-a273-b0aae3d34984"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:08.415132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.415095    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.415132 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.415127    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.415427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.415181    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.415427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.415196    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.415427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.415208    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw7j2\" (UniqueName: \"kubernetes.io/projected/e9aa2ebd-853c-45dd-a273-b0aae3d34984-kube-api-access-lw7j2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:08.491979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.491947    5419 generic.go:356] "Generic (PLEG): container finished" podID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerID="8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d" exitCode=0
Oct 31 01:44:08.492222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.491998    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"
Oct 31 01:44:08.492222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.492031    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" event={"ID":"e9aa2ebd-853c-45dd-a273-b0aae3d34984","Type":"ContainerDied","Data":"8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d"}
Oct 31 01:44:08.492222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.492096    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp" event={"ID":"e9aa2ebd-853c-45dd-a273-b0aae3d34984","Type":"ContainerDied","Data":"4b313e07ba9c4004018695b24897adb7ba9d06c1e98641a2c5526347bc801c4d"}
Oct 31 01:44:08.492222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.492129    5419 scope.go:122] "RemoveContainer" containerID="8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d"
Oct 31 01:44:08.503026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.502999    5419 scope.go:122] "RemoveContainer" containerID="467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"
Oct 31 01:44:08.512030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.512015    5419 scope.go:122] "RemoveContainer" containerID="8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d"
Oct 31 01:44:08.512320 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:08.512302    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d\": container with ID starting with 8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d not found: ID does not exist" containerID="8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d"
Oct 31 01:44:08.512409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.512332    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d"} err="failed to get container status \"8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d\": rpc error: code = NotFound desc = could not find container \"8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d\": container with ID starting with 8315e49df4dee81f2b40fd254ea749894f10bd08be064c0910ab62e1349f642d not found: ID does not exist"
Oct 31 01:44:08.512409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.512357    5419 scope.go:122] "RemoveContainer" containerID="467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"
Oct 31 01:44:08.512609 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:08.512590    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16\": container with ID starting with 467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16 not found: ID does not exist" containerID="467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"
Oct 31 01:44:08.512656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.512617    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16"} err="failed to get container status \"467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16\": rpc error: code = NotFound desc = could not find container \"467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16\": container with ID starting with 467efd199bc52fb3f9cc5a1975f1f4e9cb1f146817bced7b6d16825389b3ad16 not found: ID does not exist"
Oct 31 01:44:08.654903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.654818    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"]
Oct 31 01:44:08.658844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.658816    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df5d0b-openshift-default-7f886b9f9-hflvp"]
Oct 31 01:44:08.837211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.837181    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" path="/var/lib/kubelet/pods/2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e/volumes"
Oct 31 01:44:08.837595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.837565    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f156dc-509e-4c9b-8273-f57aec02449c" path="/var/lib/kubelet/pods/e4f156dc-509e-4c9b-8273-f57aec02449c/volumes"
Oct 31 01:44:08.837944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:08.837931    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" path="/var/lib/kubelet/pods/e9aa2ebd-853c-45dd-a273-b0aae3d34984/volumes"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.398181    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"]
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.398991    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399012    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399060    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f7b6414e-d3e2-4526-8f35-1d497320f926" containerName="revoke-keys"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399068    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b6414e-d3e2-4526-8f35-1d497320f926" containerName="revoke-keys"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399086    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" containerName="payload-pre-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399095    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" containerName="payload-pre-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399123    5419 cpu_manager.go:414] "containerMap: removing container" podUID="21ddc321-5806-498a-9ea4-38788c7d0c17" containerName="payload-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399130    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ddc321-5806-498a-9ea4-38788c7d0c17" containerName="payload-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399184    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" containerName="payload-pre-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399192    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" containerName="payload-pre-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399326    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ddc321-5806-498a-9ea4-38788c7d0c17" containerName="payload-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399340    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399350    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4a5753-43ca-4f56-96c1-bb7d84f2ba9e" containerName="payload-pre-processing"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399360    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399374    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b6414e-d3e2-4526-8f35-1d497320f926" containerName="revoke-keys"
Oct 31 01:44:13.400181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.399383    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ae76c2-4f9c-4e77-a48b-99ece7c0e409" containerName="payload-pre-processing"
Oct 31 01:44:13.404840 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.404813    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.444896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.444855    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"]
Oct 31 01:44:13.491281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.491243    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8a3796-dockercfg-wcs9c\""
Oct 31 01:44:13.491281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.491243    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8a3796\""
Oct 31 01:44:13.498767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.498735    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7qr\" (UniqueName: \"kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr\") pod \"payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.498927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.498805    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.548768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.548737    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"]
Oct 31 01:44:13.549374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.549354    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy"
Oct 31 01:44:13.549374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.549375    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9aa2ebd-853c-45dd-a273-b0aae3d34984" containerName="istio-proxy"
Oct 31 01:44:13.552839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.552820    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.600299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.600271    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7qr\" (UniqueName: \"kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr\") pod \"payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.600458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.600316    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.600458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.600345    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.600533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.600496    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wdh\" (UniqueName: \"kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh\") pod \"payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.601045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.601027    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.610407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.610380    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"]
Oct 31 01:44:13.658207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.658120    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7qr\" (UniqueName: \"kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr\") pod \"payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.701940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.701890    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.702108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.702065    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wdh\" (UniqueName: \"kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh\") pod \"payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.702605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.702586    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.721126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.721052    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:13.760805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.760767    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wdh\" (UniqueName: \"kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh\") pod \"payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:13.866934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:13.866897    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:14.098187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.098153    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"]
Oct 31 01:44:14.099393 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:14.099368    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578c5687_d459_454e_b2c2_5567165e8945.slice/crio-717e33fd2d895938e21cdb9ecd76d0c8b315d6d62728de3c3c19796ddcddaf10 WatchSource:0}: Error finding container 717e33fd2d895938e21cdb9ecd76d0c8b315d6d62728de3c3c19796ddcddaf10: Status 404 returned error can't find the container with id 717e33fd2d895938e21cdb9ecd76d0c8b315d6d62728de3c3c19796ddcddaf10
Oct 31 01:44:14.261170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.261112    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"]
Oct 31 01:44:14.537843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.537745    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9" event={"ID":"578c5687-d459-454e-b2c2-5567165e8945","Type":"ContainerStarted","Data":"9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c"}
Oct 31 01:44:14.537843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.537816    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9" event={"ID":"578c5687-d459-454e-b2c2-5567165e8945","Type":"ContainerStarted","Data":"717e33fd2d895938e21cdb9ecd76d0c8b315d6d62728de3c3c19796ddcddaf10"}
Oct 31 01:44:14.537843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.537840    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:14.538818 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.538788    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5" event={"ID":"644432a3-30e0-42eb-bf49-f53cc9a4f1b5","Type":"ContainerStarted","Data":"0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c"}
Oct 31 01:44:14.538938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.538846    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5" event={"ID":"644432a3-30e0-42eb-bf49-f53cc9a4f1b5","Type":"ContainerStarted","Data":"d37af6196688f48e9bd7990905045fff43f0f9cfd832fe2189d7e73376d7e2bb"}
Oct 31 01:44:14.538938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.538875    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:14.545867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.545846    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"]
Oct 31 01:44:14.546096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.546059    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy" containerID="cri-o://7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34" gracePeriod=30
Oct 31 01:44:14.583074 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:14.583038    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:44:14.587894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.587872    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"]
Oct 31 01:44:14.588093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.588072    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy" containerID="cri-o://7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341" gracePeriod=30
Oct 31 01:44:14.686160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.656632    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9" podStartSLOduration=1.656610533 podStartE2EDuration="1.656610533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:14.656473758 +0000 UTC m=+2375.957430349" watchObservedRunningTime="2026-10-31 01:44:14.656610533 +0000 UTC m=+2375.957567111"
Oct 31 01:44:14.914490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.914383    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5" podStartSLOduration=1.914365826 podStartE2EDuration="1.914365826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:14.912415174 +0000 UTC m=+2376.213371751" watchObservedRunningTime="2026-10-31 01:44:14.914365826 +0000 UTC m=+2376.215322404"
Oct 31 01:44:14.936302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:14.936266    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6" podUID="5288b28a-a422-49c2-aa82-8ad9879b7301" containerName="main" containerID="cri-o://5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e" gracePeriod=45
Oct 31 01:44:15.215875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.215842    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:44:15.328969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.328936    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache\") pod \"5288b28a-a422-49c2-aa82-8ad9879b7301\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") "
Oct 31 01:44:15.328969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.328975    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir\") pod \"5288b28a-a422-49c2-aa82-8ad9879b7301\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") "
Oct 31 01:44:15.329255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329028    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home\") pod \"5288b28a-a422-49c2-aa82-8ad9879b7301\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") "
Oct 31 01:44:15.329255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329051    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5\" (UniqueName: \"kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5\") pod \"5288b28a-a422-49c2-aa82-8ad9879b7301\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") "
Oct 31 01:44:15.329255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329070    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs\" (UniqueName: \"kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs\") pod \"5288b28a-a422-49c2-aa82-8ad9879b7301\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") "
Oct 31 01:44:15.329255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329252    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm\") pod \"5288b28a-a422-49c2-aa82-8ad9879b7301\" (UID: \"5288b28a-a422-49c2-aa82-8ad9879b7301\") "
Oct 31 01:44:15.329495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329258    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache" pod "5288b28a-a422-49c2-aa82-8ad9879b7301" (UID: "5288b28a-a422-49c2-aa82-8ad9879b7301"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:15.329495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329339    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home" pod "5288b28a-a422-49c2-aa82-8ad9879b7301" (UID: "5288b28a-a422-49c2-aa82-8ad9879b7301"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:15.329495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329347    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir" pod "5288b28a-a422-49c2-aa82-8ad9879b7301" (UID: "5288b28a-a422-49c2-aa82-8ad9879b7301"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:15.329664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329644    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:15.329702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329671    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:15.329702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.329686    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:15.331502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.331481    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs" pod "5288b28a-a422-49c2-aa82-8ad9879b7301" (UID: "5288b28a-a422-49c2-aa82-8ad9879b7301"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:44:15.331606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.331539    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5" pod "5288b28a-a422-49c2-aa82-8ad9879b7301" (UID: "5288b28a-a422-49c2-aa82-8ad9879b7301"). InnerVolumeSpecName "kube-api-access-zwcl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:15.331606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.331546    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm" pod "5288b28a-a422-49c2-aa82-8ad9879b7301" (UID: "5288b28a-a422-49c2-aa82-8ad9879b7301"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:15.430898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.430867    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5288b28a-a422-49c2-aa82-8ad9879b7301-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:15.430898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.430895    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcl5\" (UniqueName: \"kubernetes.io/projected/5288b28a-a422-49c2-aa82-8ad9879b7301-kube-api-access-zwcl5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:15.430898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.430905    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5288b28a-a422-49c2-aa82-8ad9879b7301-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:15.546339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.546306    5419 generic.go:356] "Generic (PLEG): container finished" podID="5288b28a-a422-49c2-aa82-8ad9879b7301" containerID="5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e" exitCode=0
Oct 31 01:44:15.546773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.546356    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"
Oct 31 01:44:15.546773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.546389    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6" event={"ID":"5288b28a-a422-49c2-aa82-8ad9879b7301","Type":"ContainerDied","Data":"5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e"}
Oct 31 01:44:15.546773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.546441    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6" event={"ID":"5288b28a-a422-49c2-aa82-8ad9879b7301","Type":"ContainerDied","Data":"1ae537e6219b915d36fb5c1291e0a0ad60f9612e0eb74e31c3c79d4a21f5bcf0"}
Oct 31 01:44:15.546773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.546457    5419 scope.go:122] "RemoveContainer" containerID="5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e"
Oct 31 01:44:15.556842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.556819    5419 scope.go:122] "RemoveContainer" containerID="5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e"
Oct 31 01:44:15.557132 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:15.557111    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e\": container with ID starting with 5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e not found: ID does not exist" containerID="5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e"
Oct 31 01:44:15.557218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.557162    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e"} err="failed to get container status \"5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e\": rpc error: code = NotFound desc = could not find container \"5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e\": container with ID starting with 5f399e5ca6ef54ea55a7a9c9e85d369cd21b7a6ff9c46ded6d2838d430b3e30e not found: ID does not exist"
Oct 31 01:44:15.709002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.708966    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"]
Oct 31 01:44:15.712255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:15.712232    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/tenantcfg-model-5e916d-kserve-6ffd9cdb7c-tbkh6"]
Oct 31 01:44:16.599713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.599680    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"]
Oct 31 01:44:16.600423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.600401    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5288b28a-a422-49c2-aa82-8ad9879b7301" containerName="main"
Oct 31 01:44:16.600515 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.600425    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5288b28a-a422-49c2-aa82-8ad9879b7301" containerName="main"
Oct 31 01:44:16.600620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.600609    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5288b28a-a422-49c2-aa82-8ad9879b7301" containerName="main"
Oct 31 01:44:16.605244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.605229    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645128    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645248    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645284    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645415    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645446    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645496    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645535    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645602    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28pnp\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.646539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.645721    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.658253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.657370    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c116f3-openshift-default-dockercfg-289mh\""
Oct 31 01:44:16.710372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.710339    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"]
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.751069    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28pnp\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.751840    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.751989    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752035    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752063    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752182    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752211    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752243    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752281    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.752757    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.753763    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.754362    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.756574    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.757762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.757023    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.764544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.763519    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.764544 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.764317    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.808733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.808700    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28pnp\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.810066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.810042    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:16.837464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.837439    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5288b28a-a422-49c2-aa82-8ad9879b7301" path="/var/lib/kubelet/pods/5288b28a-a422-49c2-aa82-8ad9879b7301/volumes"
Oct 31 01:44:16.918950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:16.918868    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:17.362636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.362608    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"]
Oct 31 01:44:17.364091 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:17.364064    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c554ef4_56d2_4e7d_bf35_69c1e5706d56.slice/crio-842c0b39dc94800a4611fb7e195442ebb2666fd81b55e4c75163583bdc25dc3c WatchSource:0}: Error finding container 842c0b39dc94800a4611fb7e195442ebb2666fd81b55e4c75163583bdc25dc3c: Status 404 returned error can't find the container with id 842c0b39dc94800a4611fb7e195442ebb2666fd81b55e4c75163583bdc25dc3c
Oct 31 01:44:17.366367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.366334    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:44:17.366448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.366399    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:44:17.366448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.366430    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:44:17.562095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.562066    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" event={"ID":"0c554ef4-56d2-4e7d-bf35-69c1e5706d56","Type":"ContainerStarted","Data":"e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"}
Oct 31 01:44:17.562582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.562124    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" event={"ID":"0c554ef4-56d2-4e7d-bf35-69c1e5706d56","Type":"ContainerStarted","Data":"842c0b39dc94800a4611fb7e195442ebb2666fd81b55e4c75163583bdc25dc3c"}
Oct 31 01:44:17.680356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.680239    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podStartSLOduration=1.680219162 podStartE2EDuration="1.680219162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:17.677401177 +0000 UTC m=+2378.978357758" watchObservedRunningTime="2026-10-31 01:44:17.680219162 +0000 UTC m=+2378.981175742"
Oct 31 01:44:17.919898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:17.919868    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:18.925405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:18.925374    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:19.576020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.575987    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:19.577012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.576992    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:44:19.892426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.892391    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:44:19.895231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.895214    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:44:19.985021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.984993    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985056    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985082    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985107    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985126    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985179    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985206    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985234    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985296    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985340    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985372    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985421    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.985458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985453    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985465    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985497    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985539    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo\") pod \"ce009443-007d-4c08-bc35-32e5fd66d4c7\" (UID: \"ce009443-007d-4c08-bc35-32e5fd66d4c7\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985585    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985671    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985710    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985742    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy\") pod \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\" (UID: \"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc\") "
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985806    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.985847    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.986266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986033    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986455    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986479    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986479    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986495    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986510    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce009443-007d-4c08-bc35-32e5fd66d4c7-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986524    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986615    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986700    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.986850 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.986782    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:19.987368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.987022    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.988253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.988226    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "kube-api-access-x7cpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:19.988533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.988483    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:19.988533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.988521    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:19.988778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.988749    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.989178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.989128    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:19.989258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.989213    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:19.989258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.989231    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h" pod "ce009443-007d-4c08-bc35-32e5fd66d4c7" (UID: "ce009443-007d-4c08-bc35-32e5fd66d4c7"). InnerVolumeSpecName "kube-api-access-2k49h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:19.989887 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:19.989869    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo" pod "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" (UID: "f1ddfed5-fee3-49a2-8ce9-baef7e9882bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:20.087483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087444    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087475    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087489    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087503    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7cpg\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-kube-api-access-x7cpg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087517    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087529    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k49h\" (UniqueName: \"kubernetes.io/projected/ce009443-007d-4c08-bc35-32e5fd66d4c7-kube-api-access-2k49h\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087540    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087552    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087564    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087578    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087590    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce009443-007d-4c08-bc35-32e5fd66d4c7-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087603    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.087760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.087633    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:20.583272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.583238    5419 generic.go:356] "Generic (PLEG): container finished" podID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerID="7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34" exitCode=0
Oct 31 01:44:20.583440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.583300    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"
Oct 31 01:44:20.583440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.583321    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" event={"ID":"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc","Type":"ContainerDied","Data":"7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34"}
Oct 31 01:44:20.583440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.583375    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx" event={"ID":"f1ddfed5-fee3-49a2-8ce9-baef7e9882bc","Type":"ContainerDied","Data":"6bd0a8620b11c95b39c3288d4d1a4a3e577c92560fd3a904f44d8ce405556303"}
Oct 31 01:44:20.583440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.583403    5419 scope.go:122] "RemoveContainer" containerID="7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34"
Oct 31 01:44:20.584844 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.584823    5419 generic.go:356] "Generic (PLEG): container finished" podID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerID="7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341" exitCode=0
Oct 31 01:44:20.584976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.584879    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"
Oct 31 01:44:20.584976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.584903    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" event={"ID":"ce009443-007d-4c08-bc35-32e5fd66d4c7","Type":"ContainerDied","Data":"7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341"}
Oct 31 01:44:20.584976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.584946    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4" event={"ID":"ce009443-007d-4c08-bc35-32e5fd66d4c7","Type":"ContainerDied","Data":"6c24302cca9e40706b66b8b746dbd6494cdc7692fcf3dd621ab524b54f63a145"}
Oct 31 01:44:20.595038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.595014    5419 scope.go:122] "RemoveContainer" containerID="c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"
Oct 31 01:44:20.608540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.608508    5419 scope.go:122] "RemoveContainer" containerID="7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34"
Oct 31 01:44:20.608871 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:20.608844    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34\": container with ID starting with 7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34 not found: ID does not exist" containerID="7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34"
Oct 31 01:44:20.608976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.608887    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34"} err="failed to get container status \"7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34\": rpc error: code = NotFound desc = could not find container \"7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34\": container with ID starting with 7369cbb32529c2c9481a0ce5d6f2ff7fb9cac6d6c99360c6fd4711b89cb4ba34 not found: ID does not exist"
Oct 31 01:44:20.608976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.608912    5419 scope.go:122] "RemoveContainer" containerID="c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"
Oct 31 01:44:20.609237 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:20.609218    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9\": container with ID starting with c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9 not found: ID does not exist" containerID="c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"
Oct 31 01:44:20.609344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.609245    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9"} err="failed to get container status \"c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9\": rpc error: code = NotFound desc = could not find container \"c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9\": container with ID starting with c9bb150dede089cbbdd0f0ec550ea03862f2bbb98dd417b3ee5c3feefa8bf5d9 not found: ID does not exist"
Oct 31 01:44:20.609344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.609263    5419 scope.go:122] "RemoveContainer" containerID="7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341"
Oct 31 01:44:20.619342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.619321    5419 scope.go:122] "RemoveContainer" containerID="7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341"
Oct 31 01:44:20.619648 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:20.619613    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341\": container with ID starting with 7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341 not found: ID does not exist" containerID="7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341"
Oct 31 01:44:20.619739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.619651    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341"} err="failed to get container status \"7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341\": rpc error: code = NotFound desc = could not find container \"7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341\": container with ID starting with 7bc244786c9c8a4ca9a33fc305b5ef39997f130cc5ce1aef11a4c53c2d197341 not found: ID does not exist"
Oct 31 01:44:20.746634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.746590    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"]
Oct 31 01:44:20.796159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.796114    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-xq5s4"]
Oct 31 01:44:20.837858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.837778    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" path="/var/lib/kubelet/pods/ce009443-007d-4c08-bc35-32e5fd66d4c7/volumes"
Oct 31 01:44:20.904474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.904444    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"]
Oct 31 01:44:20.908510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:20.908486    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8af859-openshift-default-59576db9cf-rk9gx"]
Oct 31 01:44:22.838342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:22.838309    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" path="/var/lib/kubelet/pods/f1ddfed5-fee3-49a2-8ce9-baef7e9882bc/volumes"
Oct 31 01:44:24.122505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.122473    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"]
Oct 31 01:44:24.123011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.122996    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy"
Oct 31 01:44:24.123059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123015    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy"
Oct 31 01:44:24.123059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123046    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy"
Oct 31 01:44:24.123059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123052    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy"
Oct 31 01:44:24.123188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123062    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy"
Oct 31 01:44:24.123188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123070    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy"
Oct 31 01:44:24.123188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123183    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce009443-007d-4c08-bc35-32e5fd66d4c7" containerName="istio-proxy"
Oct 31 01:44:24.123284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123198    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy"
Oct 31 01:44:24.123284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.123204    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ddfed5-fee3-49a2-8ce9-baef7e9882bc" containerName="istio-proxy"
Oct 31 01:44:24.128091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.128074    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.183277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.183238    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"]
Oct 31 01:44:24.231490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.231460    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c116f3-dockercfg-8hkpb\""
Oct 31 01:44:24.254691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.247586    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.255422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.255198    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrf8\" (UniqueName: \"kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8\") pod \"payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.255857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.232280    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c116f3\""
Oct 31 01:44:24.281172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.281120    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"]
Oct 31 01:44:24.285153 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.285122    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.341124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.341094    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"]
Oct 31 01:44:24.357151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.357094    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrf8\" (UniqueName: \"kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8\") pod \"payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.357320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.357201    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.358048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.358022    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.414877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.414810    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrf8\" (UniqueName: \"kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8\") pod \"payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.442778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.442749    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:24.458448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.458414    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrsn\" (UniqueName: \"kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn\") pod \"payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.458557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.458518    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.560103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.560071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrsn\" (UniqueName: \"kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn\") pod \"payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.560284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.560147    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.560843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.560823    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.622739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.622712    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrsn\" (UniqueName: \"kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn\") pod \"payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:24.816079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.816040    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"]
Oct 31 01:44:24.817639 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:24.817611    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e7406b_eec1_4856_88ed_50634b5b9bc8.slice/crio-1cc7eeb58d8b2632ce5d94f2f7ebf2664c0b1d4676767813ae35c798c4b144aa WatchSource:0}: Error finding container 1cc7eeb58d8b2632ce5d94f2f7ebf2664c0b1d4676767813ae35c798c4b144aa: Status 404 returned error can't find the container with id 1cc7eeb58d8b2632ce5d94f2f7ebf2664c0b1d4676767813ae35c798c4b144aa
Oct 31 01:44:24.899770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:24.899739    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:25.276479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.276449    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"]
Oct 31 01:44:25.278276 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:25.278248    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a1d415_5aa1_4e21_b60c_e72a97b2dab0.slice/crio-f18c2e1cf392cad21c12a467234d9c3296d31f64837e777bab2a6d79ea102ec0 WatchSource:0}: Error finding container f18c2e1cf392cad21c12a467234d9c3296d31f64837e777bab2a6d79ea102ec0: Status 404 returned error can't find the container with id f18c2e1cf392cad21c12a467234d9c3296d31f64837e777bab2a6d79ea102ec0
Oct 31 01:44:25.549829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.549744    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:44:25.549829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.549816    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:44:25.620424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.620381    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw" event={"ID":"76a1d415-5aa1-4e21-b60c-e72a97b2dab0","Type":"ContainerStarted","Data":"22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7"}
Oct 31 01:44:25.620596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.620441    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw" event={"ID":"76a1d415-5aa1-4e21-b60c-e72a97b2dab0","Type":"ContainerStarted","Data":"f18c2e1cf392cad21c12a467234d9c3296d31f64837e777bab2a6d79ea102ec0"}
Oct 31 01:44:25.620596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.620466    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:25.621949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.621926    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6" event={"ID":"76e7406b-eec1-4856-88ed-50634b5b9bc8","Type":"ContainerStarted","Data":"e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900"}
Oct 31 01:44:25.622066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.621976    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6" event={"ID":"76e7406b-eec1-4856-88ed-50634b5b9bc8","Type":"ContainerStarted","Data":"1cc7eeb58d8b2632ce5d94f2f7ebf2664c0b1d4676767813ae35c798c4b144aa"}
Oct 31 01:44:25.622066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.622018    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:25.890764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:25.890641    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw" podStartSLOduration=1.8906243740000002 podStartE2EDuration="1.890624374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:25.890502911 +0000 UTC m=+2387.191459523" watchObservedRunningTime="2026-10-31 01:44:25.890624374 +0000 UTC m=+2387.191580949"
Oct 31 01:44:26.009198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:26.007269    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6" podStartSLOduration=2.007249658 podStartE2EDuration="2.007249658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:26.005714998 +0000 UTC m=+2387.306671577" watchObservedRunningTime="2026-10-31 01:44:26.007249658 +0000 UTC m=+2387.308206256"
Oct 31 01:44:36.631311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:36.631278    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:44:36.631774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:36.631697    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:44:39.015499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:39.015467    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:44:43.485080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.485041    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"]
Oct 31 01:44:43.491000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.490978    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.575319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.573107    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"]
Oct 31 01:44:43.591630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591583    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nncbr\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591634    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591655    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591684    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591708    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591733    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591756    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591782    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.591893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.591812    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.605918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.605889    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c0daf4-openshift-default-dockercfg-r975r\""
Oct 31 01:44:43.694264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.694221    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.694943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.694917    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.694854    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.694961    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695262    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nncbr\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695304    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695271    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695331    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695493    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695537    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695579    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.695927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.695904    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.696187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.696164    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.696396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.696255    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.699423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.699396    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.699717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.699691    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.752539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.752464    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nncbr\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.753357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.753096    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token\") pod \"e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") " pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:43.803747 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:43.803719    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:44.240091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.240066    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"]
Oct 31 01:44:44.242633 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:44.242606    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05db315c_10b9_4a39_8656_64ed2a590be1.slice/crio-62594ecb779f211079f2931ead0a36ef7deb88a50d5014cca3833b96253b1d80 WatchSource:0}: Error finding container 62594ecb779f211079f2931ead0a36ef7deb88a50d5014cca3833b96253b1d80: Status 404 returned error can't find the container with id 62594ecb779f211079f2931ead0a36ef7deb88a50d5014cca3833b96253b1d80
Oct 31 01:44:44.245186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.245157    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:44:44.245269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.245222    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:44:44.245269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.245251    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:44:44.583502 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:44.583422    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:44:44.764242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.763304    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" event={"ID":"05db315c-10b9-4a39-8656-64ed2a590be1","Type":"ContainerStarted","Data":"e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"}
Oct 31 01:44:44.764242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.763369    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" event={"ID":"05db315c-10b9-4a39-8656-64ed2a590be1","Type":"ContainerStarted","Data":"62594ecb779f211079f2931ead0a36ef7deb88a50d5014cca3833b96253b1d80"}
Oct 31 01:44:44.804764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.804725    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:44.883337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:44.883234    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podStartSLOduration=1.8832168870000001 podStartE2EDuration="1.883216887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:44.881193567 +0000 UTC m=+2406.182150144" watchObservedRunningTime="2026-10-31 01:44:44.883216887 +0000 UTC m=+2406.184173465"
Oct 31 01:44:45.809880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:45.809848    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:46.159273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:46.159201    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"]
Oct 31 01:44:46.163957 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:46.163933    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-447154-kbcx4"]
Oct 31 01:44:46.776676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:46.776639    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:46.777511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:46.777490    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:44:46.837445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:46.837411    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b6414e-d3e2-4526-8f35-1d497320f926" path="/var/lib/kubelet/pods/f7b6414e-d3e2-4526-8f35-1d497320f926/volumes"
Oct 31 01:44:52.312173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.311310    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"]
Oct 31 01:44:52.333160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.332940    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.333160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.333063    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"]
Oct 31 01:44:52.338944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.338363    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.364584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.364511    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"]
Oct 31 01:44:52.385606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.385584    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c0daf4\""
Oct 31 01:44:52.385751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.385697    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c0daf4-dockercfg-z7xkz\""
Oct 31 01:44:52.408323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.408298    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b2d49b\""
Oct 31 01:44:52.408323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.408310    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b2d49b-dockercfg-tsrmm\""
Oct 31 01:44:52.408537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.408419    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29\") pod \"payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.408537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.408522    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.408643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.408611    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhq5z\" (UniqueName: \"kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z\") pod \"payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.408696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.408662    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.436450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.436425    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"]
Oct 31 01:44:52.509363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.509336    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhq5z\" (UniqueName: \"kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z\") pod \"payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.509529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.509387    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.509529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.509448    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29\") pod \"payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.509529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.509504    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.510204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.510178    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.510293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.510206    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.566513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.566443    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29\") pod \"payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.566661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.566525    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhq5z\" (UniqueName: \"kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z\") pod \"payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:52.666804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.666770    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:52.675028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:52.675006    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:53.055879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.054938    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"]
Oct 31 01:44:53.082710 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:53.082679    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3556719c_b6d4_4085_b84b_e88db0c6970a.slice/crio-b75019e5641335dbc354735c87a88349c6659f76766d2c512bd80f1ebca5da5b WatchSource:0}: Error finding container b75019e5641335dbc354735c87a88349c6659f76766d2c512bd80f1ebca5da5b: Status 404 returned error can't find the container with id b75019e5641335dbc354735c87a88349c6659f76766d2c512bd80f1ebca5da5b
Oct 31 01:44:53.104394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.104370    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"]
Oct 31 01:44:53.121280 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:44:53.121255    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3da6392_a65c_483e_ae2b_35ad51a4b4db.slice/crio-282b3f9ce4c91fb45c46b46c87081271e04c6d654dabf3c9f5a0ee30e1cee469 WatchSource:0}: Error finding container 282b3f9ce4c91fb45c46b46c87081271e04c6d654dabf3c9f5a0ee30e1cee469: Status 404 returned error can't find the container with id 282b3f9ce4c91fb45c46b46c87081271e04c6d654dabf3c9f5a0ee30e1cee469
Oct 31 01:44:53.322418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.322340    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"]
Oct 31 01:44:53.322857 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.322621    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy" containerID="cri-o://63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e" gracePeriod=30
Oct 31 01:44:53.842988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.842949    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" event={"ID":"3556719c-b6d4-4085-b84b-e88db0c6970a","Type":"ContainerStarted","Data":"61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528"}
Oct 31 01:44:53.843221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.843008    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" event={"ID":"3556719c-b6d4-4085-b84b-e88db0c6970a","Type":"ContainerStarted","Data":"b75019e5641335dbc354735c87a88349c6659f76766d2c512bd80f1ebca5da5b"}
Oct 31 01:44:53.843221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.843040    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:44:53.845678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.845640    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm" event={"ID":"d3da6392-a65c-483e-ae2b-35ad51a4b4db","Type":"ContainerStarted","Data":"2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541"}
Oct 31 01:44:53.845811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.845756    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm" event={"ID":"d3da6392-a65c-483e-ae2b-35ad51a4b4db","Type":"ContainerStarted","Data":"282b3f9ce4c91fb45c46b46c87081271e04c6d654dabf3c9f5a0ee30e1cee469"}
Oct 31 01:44:53.846661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.846635    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:44:53.957593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:53.957539    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" podStartSLOduration=1.957525783 podStartE2EDuration="1.957525783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:53.956004566 +0000 UTC m=+2415.256961155" watchObservedRunningTime="2026-10-31 01:44:53.957525783 +0000 UTC m=+2415.258482359"
Oct 31 01:44:58.584793 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.584768    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:44:58.698903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.698821    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.698903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.698873    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.698905    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.698997    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699047    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699090    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699128    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699175    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:58.699380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699205    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699244    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data\") pod \"caf8aa23-f61d-475f-9752-80e896ec3eab\" (UID: \"caf8aa23-f61d-475f-9752-80e896ec3eab\") "
Oct 31 01:44:58.699380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699283    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:58.699661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699567    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:44:58.699728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699666    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:58.699863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699833    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:58.699863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699845    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.699970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699864    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8aa23-f61d-475f-9752-80e896ec3eab-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.699970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699879    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.699970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.699893    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.702074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.702050    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:44:58.702189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.702070    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:44:58.702189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.702106    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:58.702189 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.702168    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k" pod "caf8aa23-f61d-475f-9752-80e896ec3eab" (UID: "caf8aa23-f61d-475f-9752-80e896ec3eab"). InnerVolumeSpecName "kube-api-access-f7s7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:44:58.801366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.801321    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.801366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.801361    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.801366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.801372    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.801672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.801381    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7s7k\" (UniqueName: \"kubernetes.io/projected/caf8aa23-f61d-475f-9752-80e896ec3eab-kube-api-access-f7s7k\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.801672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.801392    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8aa23-f61d-475f-9752-80e896ec3eab-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:44:58.802502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.802459    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm" podStartSLOduration=6.802441765 podStartE2EDuration="6.802441765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:54.080581673 +0000 UTC m=+2415.381538275" watchObservedRunningTime="2026-10-31 01:44:58.802441765 +0000 UTC m=+2420.103398342"
Oct 31 01:44:58.876966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.876929    5419 generic.go:356] "Generic (PLEG): container finished" podID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerID="63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e" exitCode=0
Oct 31 01:44:58.877159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.876994    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"
Oct 31 01:44:58.877159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.877037    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" event={"ID":"caf8aa23-f61d-475f-9752-80e896ec3eab","Type":"ContainerDied","Data":"63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e"}
Oct 31 01:44:58.877159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.877075    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v" event={"ID":"caf8aa23-f61d-475f-9752-80e896ec3eab","Type":"ContainerDied","Data":"2b5b0cc052e03d84b7787d33882cf72e1f15156dcfcea724a0cf0b02276a1966"}
Oct 31 01:44:58.877159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.877094    5419 scope.go:122] "RemoveContainer" containerID="63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e"
Oct 31 01:44:58.887354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.887334    5419 scope.go:122] "RemoveContainer" containerID="ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"
Oct 31 01:44:58.897325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.897308    5419 scope.go:122] "RemoveContainer" containerID="63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e"
Oct 31 01:44:58.897618 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:58.897598    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e\": container with ID starting with 63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e not found: ID does not exist" containerID="63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e"
Oct 31 01:44:58.897664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.897629    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e"} err="failed to get container status \"63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e\": rpc error: code = NotFound desc = could not find container \"63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e\": container with ID starting with 63d49b45f3b5b949fc1bf0041861812b6a14799a023fe108176193f0f327964e not found: ID does not exist"
Oct 31 01:44:58.897664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.897647    5419 scope.go:122] "RemoveContainer" containerID="ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"
Oct 31 01:44:58.897905 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:44:58.897882    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee\": container with ID starting with ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee not found: ID does not exist" containerID="ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"
Oct 31 01:44:58.897960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:58.897912    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee"} err="failed to get container status \"ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee\": rpc error: code = NotFound desc = could not find container \"ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee\": container with ID starting with ee7aef80e5fb1b84561da3bc8d732b3d5476d25883aecbdca18ed7310d0fceee not found: ID does not exist"
Oct 31 01:44:59.041387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:59.041351    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"]
Oct 31 01:44:59.047198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:44:59.047173    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-447154-openshift-default-6584c45855-n4z4v"]
Oct 31 01:45:00.195752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.195585    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"]
Oct 31 01:45:00.196609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.196580    5419 cpu_manager.go:414] "containerMap: removing container" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy"
Oct 31 01:45:00.196609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.196605    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy"
Oct 31 01:45:00.196859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.196771    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy"
Oct 31 01:45:00.196859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.196784    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy"
Oct 31 01:45:00.202317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.202294    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.215864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.215838    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"]
Oct 31 01:45:00.216882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.216860    5419 cpu_manager.go:414] "containerMap: removing container" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy"
Oct 31 01:45:00.217038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.217011    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" containerName="istio-proxy"
Oct 31 01:45:00.220929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.220911    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.221049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.220986    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"]
Oct 31 01:45:00.225656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.225634    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.248660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.246047    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"]
Oct 31 01:45:00.250498 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.250473    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.269097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.269067    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:45:00.269364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.269310    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:45:00.292182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.291642    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:45:00.294814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.294766    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"]
Oct 31 01:45:00.299728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.299706    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.319086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319053    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.319254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319221    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.319320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319259    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.319391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319345    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4wc\" (UniqueName: \"kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.319453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319409    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrmr\" (UniqueName: \"kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.319509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319462    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7zg\" (UniqueName: \"kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg\") pod \"maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.319570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319510    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.319623 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.319565    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgf8\" (UniqueName: \"kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.320514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.320496    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"]
Oct 31 01:45:00.324450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.324434    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"]
Oct 31 01:45:00.324541 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.324517    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.327769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.327748    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"]
Oct 31 01:45:00.331181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.330983    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"]
Oct 31 01:45:00.369011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.368981    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"]
Oct 31 01:45:00.378361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.378332    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"]
Oct 31 01:45:00.391175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.390024    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"]
Oct 31 01:45:00.428661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428620    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.428858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428686    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.428858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428732    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.428858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428775    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.428858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428831    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4wc\" (UniqueName: \"kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.429086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428888    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sbqg\" (UniqueName: \"kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.429086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.428947    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrmr\" (UniqueName: \"kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.429086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.429008    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7zg\" (UniqueName: \"kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg\") pod \"maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.429086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.429079    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.429314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.429129    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2sb\" (UniqueName: \"kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb\") pod \"maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.429314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.429189    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.429314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.429226    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgf8\" (UniqueName: \"kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.430595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.430561    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.431299 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.431270    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.432848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.432821    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.434664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.434626    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.488473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.488443    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgf8\" (UniqueName: \"kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.497026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.496993    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4wc\" (UniqueName: \"kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc\") pod \"maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.497217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.497076    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7zg\" (UniqueName: \"kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg\") pod \"maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.497320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.497297    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrmr\" (UniqueName: \"kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.518039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.518013    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:00.530427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.530404    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sbqg\" (UniqueName: \"kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.530546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.530518    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2sb\" (UniqueName: \"kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb\") pod \"maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.530606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.530551    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.530671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.530619    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.531278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.531256    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.531418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.531393    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.540169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.540117    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:00.550272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.550251    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:00.568354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.568328    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:00.589013 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.588976    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2sb\" (UniqueName: \"kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb\") pod \"maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.590114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.590003    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sbqg\" (UniqueName: \"kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.621087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.621060    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:00.638274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.638128    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:00.858994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.858952    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf8aa23-f61d-475f-9752-80e896ec3eab" path="/var/lib/kubelet/pods/caf8aa23-f61d-475f-9752-80e896ec3eab/volumes"
Oct 31 01:45:00.910416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.910386    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"]
Oct 31 01:45:00.911518 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:45:00.911483    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370a984f_c52e_4f8e_aea3_5667060334a4.slice/crio-5f72cfe6db380d3c4f40ac855d2e5525350a9e10dfef6b3730136721f907ed8c WatchSource:0}: Error finding container 5f72cfe6db380d3c4f40ac855d2e5525350a9e10dfef6b3730136721f907ed8c: Status 404 returned error can't find the container with id 5f72cfe6db380d3c4f40ac855d2e5525350a9e10dfef6b3730136721f907ed8c
Oct 31 01:45:00.958575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.958554    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"]
Oct 31 01:45:00.959807 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:45:00.959786    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cb349e_660d_4ff5_84f7_aa2f64de328c.slice/crio-4546b6ff35ab54e7bf24c8dec3c1ef083ba4dab88b0a0dad24aaacd335a41450 WatchSource:0}: Error finding container 4546b6ff35ab54e7bf24c8dec3c1ef083ba4dab88b0a0dad24aaacd335a41450: Status 404 returned error can't find the container with id 4546b6ff35ab54e7bf24c8dec3c1ef083ba4dab88b0a0dad24aaacd335a41450
Oct 31 01:45:00.987573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:00.987549    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"]
Oct 31 01:45:00.988390 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:45:00.988361    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2143d0_5d54_4329_a590_c855394930ea.slice/crio-24bc2617aad2b7b06f121d04f5cac37d7f3cd1bda31817b59e89756326f3c343 WatchSource:0}: Error finding container 24bc2617aad2b7b06f121d04f5cac37d7f3cd1bda31817b59e89756326f3c343: Status 404 returned error can't find the container with id 24bc2617aad2b7b06f121d04f5cac37d7f3cd1bda31817b59e89756326f3c343
Oct 31 01:45:01.008916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.008894    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"]
Oct 31 01:45:01.020373 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:45:01.020348    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76772c3b_0c58_4ddd_a78f_1f91b09b0706.slice/crio-0106516425a0ce68bb5c93f35436791843ca46c2409f236c9ddc773912feb565 WatchSource:0}: Error finding container 0106516425a0ce68bb5c93f35436791843ca46c2409f236c9ddc773912feb565: Status 404 returned error can't find the container with id 0106516425a0ce68bb5c93f35436791843ca46c2409f236c9ddc773912feb565
Oct 31 01:45:01.091866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.091834    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"]
Oct 31 01:45:01.094310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.094289    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"]
Oct 31 01:45:01.179098 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:45:01.179069    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4338a5_329d_43a1_a0ef_b4c7d427b5e7.slice/crio-951b6db04a4c74970d93f4f46b98520e826b27402ef24d092de2849ca244ef2b WatchSource:0}: Error finding container 951b6db04a4c74970d93f4f46b98520e826b27402ef24d092de2849ca244ef2b: Status 404 returned error can't find the container with id 951b6db04a4c74970d93f4f46b98520e826b27402ef24d092de2849ca244ef2b
Oct 31 01:45:01.180167 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:45:01.180079    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3afae06_e4c7_4fc5_b6e0_c98cf82aa7e6.slice/crio-f8b2354799637137f242ee9ab671e1a93f292be390a050bcb964f48515d03cf0 WatchSource:0}: Error finding container f8b2354799637137f242ee9ab671e1a93f292be390a050bcb964f48515d03cf0: Status 404 returned error can't find the container with id f8b2354799637137f242ee9ab671e1a93f292be390a050bcb964f48515d03cf0
Oct 31 01:45:01.901704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.901673    5419 generic.go:356] "Generic (PLEG): container finished" podID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerID="78bc999f5a3cb3c07e0894ea67a28ebaf651bfea50c0a5ab034403902e34c4da" exitCode=60
Oct 31 01:45:01.902081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.901760    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerDied","Data":"78bc999f5a3cb3c07e0894ea67a28ebaf651bfea50c0a5ab034403902e34c4da"}
Oct 31 01:45:01.902081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.901818    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerStarted","Data":"0106516425a0ce68bb5c93f35436791843ca46c2409f236c9ddc773912feb565"}
Oct 31 01:45:01.902192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.902103    5419 scope.go:122] "RemoveContainer" containerID="78bc999f5a3cb3c07e0894ea67a28ebaf651bfea50c0a5ab034403902e34c4da"
Oct 31 01:45:01.904355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.904330    5419 generic.go:356] "Generic (PLEG): container finished" podID="370a984f-c52e-4f8e-aea3-5667060334a4" containerID="6b1a00f663981e1cf38880b62f699b46664eaa187523df2a30e3f5d95e127717" exitCode=60
Oct 31 01:45:01.904460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.904431    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" event={"ID":"370a984f-c52e-4f8e-aea3-5667060334a4","Type":"ContainerDied","Data":"6b1a00f663981e1cf38880b62f699b46664eaa187523df2a30e3f5d95e127717"}
Oct 31 01:45:01.904523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.904469    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" event={"ID":"370a984f-c52e-4f8e-aea3-5667060334a4","Type":"ContainerStarted","Data":"5f72cfe6db380d3c4f40ac855d2e5525350a9e10dfef6b3730136721f907ed8c"}
Oct 31 01:45:01.904661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.904638    5419 scope.go:122] "RemoveContainer" containerID="6b1a00f663981e1cf38880b62f699b46664eaa187523df2a30e3f5d95e127717"
Oct 31 01:45:01.906447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.906419    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerStarted","Data":"183323ab0b48cea8c66aeb9d4a29c9f6b4ced4c79f2a3640954f8159ac169e63"}
Oct 31 01:45:01.906549 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.906474    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerStarted","Data":"f8b2354799637137f242ee9ab671e1a93f292be390a050bcb964f48515d03cf0"}
Oct 31 01:45:01.908193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.908174    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerStarted","Data":"a0106fe63e7e0c3e7cca8bdb19eba6e511dc49a0c58d3ce533110a86ef488aaf"}
Oct 31 01:45:01.908272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.908212    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerStarted","Data":"24bc2617aad2b7b06f121d04f5cac37d7f3cd1bda31817b59e89756326f3c343"}
Oct 31 01:45:01.909864 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.909838    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerStarted","Data":"df77054833bc059873d8c077abeda13017ba4ae04a81a103ff46425e67783560"}
Oct 31 01:45:01.909942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.909884    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerStarted","Data":"951b6db04a4c74970d93f4f46b98520e826b27402ef24d092de2849ca244ef2b"}
Oct 31 01:45:01.911465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.911440    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerStarted","Data":"0f05465b99585e26e815f8aaade70d6a4b9ea6b917bf3bfb5ae53587084c0d96"}
Oct 31 01:45:01.911584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:01.911484    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerStarted","Data":"4546b6ff35ab54e7bf24c8dec3c1ef083ba4dab88b0a0dad24aaacd335a41450"}
Oct 31 01:45:02.132987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.132905    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" podStartSLOduration=2.132891631 podStartE2EDuration="2.132891631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.131828266 +0000 UTC m=+2423.432784843" watchObservedRunningTime="2026-10-31 01:45:02.132891631 +0000 UTC m=+2423.433848207"
Oct 31 01:45:02.245594 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.245547    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" podStartSLOduration=2.245530671 podStartE2EDuration="2.245530671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.244489695 +0000 UTC m=+2423.545446274" watchObservedRunningTime="2026-10-31 01:45:02.245530671 +0000 UTC m=+2423.546487247"
Oct 31 01:45:02.471668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.471602    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" podStartSLOduration=2.471587295 podStartE2EDuration="2.471587295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.46938544 +0000 UTC m=+2423.770342029" watchObservedRunningTime="2026-10-31 01:45:02.471587295 +0000 UTC m=+2423.772543871"
Oct 31 01:45:02.582535 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.582481    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" podStartSLOduration=2.582465858 podStartE2EDuration="2.582465858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.581036731 +0000 UTC m=+2423.881993300" watchObservedRunningTime="2026-10-31 01:45:02.582465858 +0000 UTC m=+2423.883422435"
Oct 31 01:45:02.921286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.921251    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerStarted","Data":"bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03"}
Oct 31 01:45:02.924031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.923976    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" event={"ID":"370a984f-c52e-4f8e-aea3-5667060334a4","Type":"ContainerStarted","Data":"1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb"}
Oct 31 01:45:02.926750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.926725    5419 generic.go:356] "Generic (PLEG): container finished" podID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerID="183323ab0b48cea8c66aeb9d4a29c9f6b4ced4c79f2a3640954f8159ac169e63" exitCode=60
Oct 31 01:45:02.926882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.926800    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerDied","Data":"183323ab0b48cea8c66aeb9d4a29c9f6b4ced4c79f2a3640954f8159ac169e63"}
Oct 31 01:45:02.927487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.927048    5419 scope.go:122] "RemoveContainer" containerID="183323ab0b48cea8c66aeb9d4a29c9f6b4ced4c79f2a3640954f8159ac169e63"
Oct 31 01:45:02.928559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.928536    5419 generic.go:356] "Generic (PLEG): container finished" podID="8f2143d0-5d54-4329-a590-c855394930ea" containerID="a0106fe63e7e0c3e7cca8bdb19eba6e511dc49a0c58d3ce533110a86ef488aaf" exitCode=60
Oct 31 01:45:02.928664 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.928587    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerDied","Data":"a0106fe63e7e0c3e7cca8bdb19eba6e511dc49a0c58d3ce533110a86ef488aaf"}
Oct 31 01:45:02.928878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.928852    5419 scope.go:122] "RemoveContainer" containerID="a0106fe63e7e0c3e7cca8bdb19eba6e511dc49a0c58d3ce533110a86ef488aaf"
Oct 31 01:45:02.930373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.930350    5419 generic.go:356] "Generic (PLEG): container finished" podID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerID="df77054833bc059873d8c077abeda13017ba4ae04a81a103ff46425e67783560" exitCode=60
Oct 31 01:45:02.930468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.930421    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerDied","Data":"df77054833bc059873d8c077abeda13017ba4ae04a81a103ff46425e67783560"}
Oct 31 01:45:02.930614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.930586    5419 scope.go:122] "RemoveContainer" containerID="df77054833bc059873d8c077abeda13017ba4ae04a81a103ff46425e67783560"
Oct 31 01:45:02.931985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.931965    5419 generic.go:356] "Generic (PLEG): container finished" podID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerID="0f05465b99585e26e815f8aaade70d6a4b9ea6b917bf3bfb5ae53587084c0d96" exitCode=60
Oct 31 01:45:02.932086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.932019    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerDied","Data":"0f05465b99585e26e815f8aaade70d6a4b9ea6b917bf3bfb5ae53587084c0d96"}
Oct 31 01:45:02.932270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:02.932252    5419 scope.go:122] "RemoveContainer" containerID="0f05465b99585e26e815f8aaade70d6a4b9ea6b917bf3bfb5ae53587084c0d96"
Oct 31 01:45:03.040125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.040069    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" podStartSLOduration=3.040054918 podStartE2EDuration="3.040054918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.037919845 +0000 UTC m=+2424.338876439" watchObservedRunningTime="2026-10-31 01:45:03.040054918 +0000 UTC m=+2424.341011494"
Oct 31 01:45:03.287075 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:03.287046    5419 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3afae06_e4c7_4fc5_b6e0_c98cf82aa7e6.slice/crio-710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:45:03.394682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.394597    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" podStartSLOduration=3.394575183 podStartE2EDuration="3.394575183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.389407434 +0000 UTC m=+2424.690364021" watchObservedRunningTime="2026-10-31 01:45:03.394575183 +0000 UTC m=+2424.695531802"
Oct 31 01:45:03.939863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.939832    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerStarted","Data":"710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454"}
Oct 31 01:45:03.941825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.941795    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerStarted","Data":"f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be"}
Oct 31 01:45:03.943636 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.943606    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerStarted","Data":"a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2"}
Oct 31 01:45:03.945543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.945520    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerStarted","Data":"ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef"}
Oct 31 01:45:03.947379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.947354    5419 generic.go:356] "Generic (PLEG): container finished" podID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerID="bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03" exitCode=60
Oct 31 01:45:03.947486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.947423    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerDied","Data":"bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03"}
Oct 31 01:45:03.947486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.947469    5419 scope.go:122] "RemoveContainer" containerID="78bc999f5a3cb3c07e0894ea67a28ebaf651bfea50c0a5ab034403902e34c4da"
Oct 31 01:45:03.947658 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.947640    5419 scope.go:122] "RemoveContainer" containerID="bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03"
Oct 31 01:45:03.947917 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:03.947897    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph_odh-ai-gateway-infra(76772c3b-0c58-4ddd-a78f-1f91b09b0706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706"
Oct 31 01:45:03.949330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.949308    5419 generic.go:356] "Generic (PLEG): container finished" podID="370a984f-c52e-4f8e-aea3-5667060334a4" containerID="1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb" exitCode=60
Oct 31 01:45:03.949414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.949347    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" event={"ID":"370a984f-c52e-4f8e-aea3-5667060334a4","Type":"ContainerDied","Data":"1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb"}
Oct 31 01:45:03.949541 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.949527    5419 scope.go:122] "RemoveContainer" containerID="1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb"
Oct 31 01:45:03.949765 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:03.949748    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr_odh-ai-gateway-infra(370a984f-c52e-4f8e-aea3-5667060334a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" podUID="370a984f-c52e-4f8e-aea3-5667060334a4"
Oct 31 01:45:03.960059 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:03.960042    5419 scope.go:122] "RemoveContainer" containerID="6b1a00f663981e1cf38880b62f699b46664eaa187523df2a30e3f5d95e127717"
Oct 31 01:45:04.853704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.853678    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:45:04.958684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.958646    5419 generic.go:356] "Generic (PLEG): container finished" podID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerID="a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2" exitCode=60
Oct 31 01:45:04.959222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.958689    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerDied","Data":"a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2"}
Oct 31 01:45:04.959222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.958758    5419 scope.go:122] "RemoveContainer" containerID="df77054833bc059873d8c077abeda13017ba4ae04a81a103ff46425e67783560"
Oct 31 01:45:04.959222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.959073    5419 scope.go:122] "RemoveContainer" containerID="a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2"
Oct 31 01:45:04.959440 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:04.959405    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h_odh-ai-gateway-infra(3c4338a5-329d-43a1-a0ef-b4c7d427b5e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7"
Oct 31 01:45:04.961124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.961099    5419 generic.go:356] "Generic (PLEG): container finished" podID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerID="ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef" exitCode=60
Oct 31 01:45:04.961285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.961185    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerDied","Data":"ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef"}
Oct 31 01:45:04.961509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.961490    5419 scope.go:122] "RemoveContainer" containerID="ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef"
Oct 31 01:45:04.961786 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:04.961764    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6_odh-ai-gateway-infra(e7cb349e-660d-4ff5-84f7-aa2f64de328c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c"
Oct 31 01:45:04.963674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.963651    5419 scope.go:122] "RemoveContainer" containerID="bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03"
Oct 31 01:45:04.963955 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:04.963930    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph_odh-ai-gateway-infra(76772c3b-0c58-4ddd-a78f-1f91b09b0706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706"
Oct 31 01:45:04.965317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.965297    5419 scope.go:122] "RemoveContainer" containerID="1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb"
Oct 31 01:45:04.965604 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:04.965573    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr_odh-ai-gateway-infra(370a984f-c52e-4f8e-aea3-5667060334a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" podUID="370a984f-c52e-4f8e-aea3-5667060334a4"
Oct 31 01:45:04.967680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.967658    5419 generic.go:356] "Generic (PLEG): container finished" podID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerID="710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454" exitCode=60
Oct 31 01:45:04.967805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.967731    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerDied","Data":"710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454"}
Oct 31 01:45:04.968045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.968025    5419 scope.go:122] "RemoveContainer" containerID="710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454"
Oct 31 01:45:04.968316 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:04.968290    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc_odh-ai-gateway-infra(c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6"
Oct 31 01:45:04.969806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.969744    5419 generic.go:356] "Generic (PLEG): container finished" podID="8f2143d0-5d54-4329-a590-c855394930ea" containerID="f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be" exitCode=60
Oct 31 01:45:04.969908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.969844    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerDied","Data":"f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be"}
Oct 31 01:45:04.970103 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.970086    5419 scope.go:122] "RemoveContainer" containerID="f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be"
Oct 31 01:45:04.970390 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:04.970359    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv_odh-ai-gateway-infra(8f2143d0-5d54-4329-a590-c855394930ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" podUID="8f2143d0-5d54-4329-a590-c855394930ea"
Oct 31 01:45:04.974954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.974937    5419 scope.go:122] "RemoveContainer" containerID="0f05465b99585e26e815f8aaade70d6a4b9ea6b917bf3bfb5ae53587084c0d96"
Oct 31 01:45:04.993988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:04.993969    5419 scope.go:122] "RemoveContainer" containerID="183323ab0b48cea8c66aeb9d4a29c9f6b4ced4c79f2a3640954f8159ac169e63"
Oct 31 01:45:05.013588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:05.013567    5419 scope.go:122] "RemoveContainer" containerID="a0106fe63e7e0c3e7cca8bdb19eba6e511dc49a0c58d3ce533110a86ef488aaf"
Oct 31 01:45:05.859218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:05.859189    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:45:05.980813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:05.980786    5419 scope.go:122] "RemoveContainer" containerID="a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2"
Oct 31 01:45:05.981324 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:05.981132    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h_odh-ai-gateway-infra(3c4338a5-329d-43a1-a0ef-b4c7d427b5e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7"
Oct 31 01:45:05.982971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:05.982947    5419 scope.go:122] "RemoveContainer" containerID="ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef"
Oct 31 01:45:05.983265 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:05.983236    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6_odh-ai-gateway-infra(e7cb349e-660d-4ff5-84f7-aa2f64de328c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c"
Oct 31 01:45:05.984781 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:05.984760    5419 scope.go:122] "RemoveContainer" containerID="710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454"
Oct 31 01:45:05.984967 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:05.984944    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc_odh-ai-gateway-infra(c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6"
Oct 31 01:45:05.987101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:05.987083    5419 scope.go:122] "RemoveContainer" containerID="f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be"
Oct 31 01:45:05.987318 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:05.987298    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv_odh-ai-gateway-infra(8f2143d0-5d54-4329-a590-c855394930ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" podUID="8f2143d0-5d54-4329-a590-c855394930ea"
Oct 31 01:45:13.831196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:13.831132    5419 scope.go:122] "RemoveContainer" containerID="bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03"
Oct 31 01:45:14.038863 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:14.038814    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerStarted","Data":"a1040d8259486aa7f3182cdfed5242f4040c0a6b30852e398011470cdc08269e"}
Oct 31 01:45:14.582265 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:14.582230    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:45:14.830960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:14.830884    5419 scope.go:122] "RemoveContainer" containerID="1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb"
Oct 31 01:45:15.047460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:15.047430    5419 generic.go:356] "Generic (PLEG): container finished" podID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerID="a1040d8259486aa7f3182cdfed5242f4040c0a6b30852e398011470cdc08269e" exitCode=60
Oct 31 01:45:15.047865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:15.047503    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerDied","Data":"a1040d8259486aa7f3182cdfed5242f4040c0a6b30852e398011470cdc08269e"}
Oct 31 01:45:15.047865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:15.047570    5419 scope.go:122] "RemoveContainer" containerID="bbdbf175d988e946a6749cea72020c9b27ccb1e193346b349e9283920046bd03"
Oct 31 01:45:15.047993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:15.047956    5419 scope.go:122] "RemoveContainer" containerID="a1040d8259486aa7f3182cdfed5242f4040c0a6b30852e398011470cdc08269e"
Oct 31 01:45:15.048287 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:15.048264    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph_odh-ai-gateway-infra(76772c3b-0c58-4ddd-a78f-1f91b09b0706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706"
Oct 31 01:45:15.166480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:15.166453    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"]
Oct 31 01:45:16.059071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.059041    5419 generic.go:356] "Generic (PLEG): container finished" podID="370a984f-c52e-4f8e-aea3-5667060334a4" containerID="eea103dcf6e20187ad1f2702362436dd17782429388c328e4d99202e3ff16a1a" exitCode=60
Oct 31 01:45:16.059481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.059107    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" event={"ID":"370a984f-c52e-4f8e-aea3-5667060334a4","Type":"ContainerDied","Data":"eea103dcf6e20187ad1f2702362436dd17782429388c328e4d99202e3ff16a1a"}
Oct 31 01:45:16.059481 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.059218    5419 scope.go:122] "RemoveContainer" containerID="1d437ecbb119b63cca49c52759cf6ca6b31383cf4122c2e5a2b338872c5c91bb"
Oct 31 01:45:16.059586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.059567    5419 scope.go:122] "RemoveContainer" containerID="eea103dcf6e20187ad1f2702362436dd17782429388c328e4d99202e3ff16a1a"
Oct 31 01:45:16.059927 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:16.059907    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr_odh-ai-gateway-infra(370a984f-c52e-4f8e-aea3-5667060334a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" podUID="370a984f-c52e-4f8e-aea3-5667060334a4"
Oct 31 01:45:16.207684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.207660    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:16.319429 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.319332    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8\" (UniqueName: \"kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8\") pod \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") "
Oct 31 01:45:16.319625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.319497    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca\") pod \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\" (UID: \"76772c3b-0c58-4ddd-a78f-1f91b09b0706\") "
Oct 31 01:45:16.319867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.319844    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca" pod "76772c3b-0c58-4ddd-a78f-1f91b09b0706" (UID: "76772c3b-0c58-4ddd-a78f-1f91b09b0706"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:16.321624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.321600    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8" pod "76772c3b-0c58-4ddd-a78f-1f91b09b0706" (UID: "76772c3b-0c58-4ddd-a78f-1f91b09b0706"). InnerVolumeSpecName "kube-api-access-wxgf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:16.420748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.420716    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxgf8\" (UniqueName: \"kubernetes.io/projected/76772c3b-0c58-4ddd-a78f-1f91b09b0706-kube-api-access-wxgf8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:16.420748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.420748    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76772c3b-0c58-4ddd-a78f-1f91b09b0706-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:16.830758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.830727    5419 scope.go:122] "RemoveContainer" containerID="ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef"
Oct 31 01:45:16.830963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.830826    5419 scope.go:122] "RemoveContainer" containerID="a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2"
Oct 31 01:45:16.830963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:16.830891    5419 scope.go:122] "RemoveContainer" containerID="710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454"
Oct 31 01:45:17.067625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.067582    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"
Oct 31 01:45:17.068114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.067583    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph" event={"ID":"76772c3b-0c58-4ddd-a78f-1f91b09b0706","Type":"ContainerDied","Data":"0106516425a0ce68bb5c93f35436791843ca46c2409f236c9ddc773912feb565"}
Oct 31 01:45:17.068114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.067738    5419 scope.go:122] "RemoveContainer" containerID="a1040d8259486aa7f3182cdfed5242f4040c0a6b30852e398011470cdc08269e"
Oct 31 01:45:17.232970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.232943    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"]
Oct 31 01:45:17.282073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.282051    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890185-nmxph"]
Oct 31 01:45:17.299814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.299789    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"]
Oct 31 01:45:17.500675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.500651    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:17.532696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.532673    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr\" (UniqueName: \"kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr\") pod \"370a984f-c52e-4f8e-aea3-5667060334a4\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") "
Oct 31 01:45:17.532879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.532853    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca\") pod \"370a984f-c52e-4f8e-aea3-5667060334a4\" (UID: \"370a984f-c52e-4f8e-aea3-5667060334a4\") "
Oct 31 01:45:17.533161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.533121    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca" pod "370a984f-c52e-4f8e-aea3-5667060334a4" (UID: "370a984f-c52e-4f8e-aea3-5667060334a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:17.533298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.533281    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370a984f-c52e-4f8e-aea3-5667060334a4-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:17.534814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.534789    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr" pod "370a984f-c52e-4f8e-aea3-5667060334a4" (UID: "370a984f-c52e-4f8e-aea3-5667060334a4"). InnerVolumeSpecName "kube-api-access-rtrmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:17.634702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:17.634619    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrmr\" (UniqueName: \"kubernetes.io/projected/370a984f-c52e-4f8e-aea3-5667060334a4-kube-api-access-rtrmr\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:18.078377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.078347    5419 generic.go:356] "Generic (PLEG): container finished" podID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerID="7f2b23ad28b85bd24aed1af5ad2fecc0e959cb81a1a1deb4a9ad0004c5089354" exitCode=60
Oct 31 01:45:18.078838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.078432    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerDied","Data":"7f2b23ad28b85bd24aed1af5ad2fecc0e959cb81a1a1deb4a9ad0004c5089354"}
Oct 31 01:45:18.078838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.078510    5419 scope.go:122] "RemoveContainer" containerID="a69f63ed7a73efcb59545c7c69102475466c5505403b4df6813dd729f2c9fdb2"
Oct 31 01:45:18.078974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.078905    5419 scope.go:122] "RemoveContainer" containerID="7f2b23ad28b85bd24aed1af5ad2fecc0e959cb81a1a1deb4a9ad0004c5089354"
Oct 31 01:45:18.079240 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:18.079216    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h_odh-ai-gateway-infra(3c4338a5-329d-43a1-a0ef-b4c7d427b5e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7"
Oct 31 01:45:18.080585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.080567    5419 generic.go:356] "Generic (PLEG): container finished" podID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerID="763f905c17463d82a8ce81d15959d47f848fdcc75f5fa45afc52ef63d9a2c3bd" exitCode=60
Oct 31 01:45:18.080706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.080642    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerDied","Data":"763f905c17463d82a8ce81d15959d47f848fdcc75f5fa45afc52ef63d9a2c3bd"}
Oct 31 01:45:18.081065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.081022    5419 scope.go:122] "RemoveContainer" containerID="763f905c17463d82a8ce81d15959d47f848fdcc75f5fa45afc52ef63d9a2c3bd"
Oct 31 01:45:18.081343 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:18.081321    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6_odh-ai-gateway-infra(e7cb349e-660d-4ff5-84f7-aa2f64de328c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c"
Oct 31 01:45:18.083034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.083014    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"
Oct 31 01:45:18.083178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.083042    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr" event={"ID":"370a984f-c52e-4f8e-aea3-5667060334a4","Type":"ContainerDied","Data":"5f72cfe6db380d3c4f40ac855d2e5525350a9e10dfef6b3730136721f907ed8c"}
Oct 31 01:45:18.084905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.084882    5419 generic.go:356] "Generic (PLEG): container finished" podID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerID="68dd0b9e0c0f0b29cedfd39bb92743015fd4129238b046029972f52fd0a9b762" exitCode=60
Oct 31 01:45:18.085005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.084958    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerDied","Data":"68dd0b9e0c0f0b29cedfd39bb92743015fd4129238b046029972f52fd0a9b762"}
Oct 31 01:45:18.085357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.085342    5419 scope.go:122] "RemoveContainer" containerID="68dd0b9e0c0f0b29cedfd39bb92743015fd4129238b046029972f52fd0a9b762"
Oct 31 01:45:18.085566 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:18.085551    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc_odh-ai-gateway-infra(c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6"
Oct 31 01:45:18.094965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.094946    5419 scope.go:122] "RemoveContainer" containerID="ec68feccaef4813ddbccf3555cb69af5f3982461b46e1c5d07649a0aa2f28cef"
Oct 31 01:45:18.108450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.108426    5419 scope.go:122] "RemoveContainer" containerID="eea103dcf6e20187ad1f2702362436dd17782429388c328e4d99202e3ff16a1a"
Oct 31 01:45:18.117120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.117100    5419 scope.go:122] "RemoveContainer" containerID="710a91db205f0991327a7cb4a6fe85446ebf1bc132dbfa70dad9976fbb428454"
Oct 31 01:45:18.531528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.530425    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"]
Oct 31 01:45:18.534817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.534785    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-71bc98-29890185-s6qxr"]
Oct 31 01:45:18.833960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.833932    5419 scope.go:122] "RemoveContainer" containerID="f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be"
Oct 31 01:45:18.838487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.838465    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" path="/var/lib/kubelet/pods/370a984f-c52e-4f8e-aea3-5667060334a4/volumes"
Oct 31 01:45:18.838869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:18.838857    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" path="/var/lib/kubelet/pods/76772c3b-0c58-4ddd-a78f-1f91b09b0706/volumes"
Oct 31 01:45:19.202048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.202010    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"]
Oct 31 01:45:19.318175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.318125    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"]
Oct 31 01:45:19.342951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.342868    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:19.428918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.428888    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"]
Oct 31 01:45:19.457320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.457118    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg\" (UniqueName: \"kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg\") pod \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") "
Oct 31 01:45:19.457448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.457408    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca\") pod \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\" (UID: \"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7\") "
Oct 31 01:45:19.458192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.458168    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca" pod "3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" (UID: "3c4338a5-329d-43a1-a0ef-b4c7d427b5e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.459842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.459810    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg" pod "3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" (UID: "3c4338a5-329d-43a1-a0ef-b4c7d427b5e7"). InnerVolumeSpecName "kube-api-access-2sbqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.477034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.477012    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:19.558861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.558795    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb\" (UniqueName: \"kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb\") pod \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") "
Oct 31 01:45:19.559017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.558961    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca\") pod \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\" (UID: \"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6\") "
Oct 31 01:45:19.559355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.559332    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca" pod "c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" (UID: "c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.559582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.559562    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.559646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.559585    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.559646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.559599    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sbqg\" (UniqueName: \"kubernetes.io/projected/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7-kube-api-access-2sbqg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.561598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.561563    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb" pod "c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" (UID: "c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6"). InnerVolumeSpecName "kube-api-access-fn2sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.580458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.580437    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:19.661347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.661312    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc\" (UniqueName: \"kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc\") pod \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") "
Oct 31 01:45:19.661573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.661554    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca\") pod \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\" (UID: \"e7cb349e-660d-4ff5-84f7-aa2f64de328c\") "
Oct 31 01:45:19.662035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.662009    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn2sb\" (UniqueName: \"kubernetes.io/projected/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6-kube-api-access-fn2sb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.663786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.663757    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca" pod "e7cb349e-660d-4ff5-84f7-aa2f64de328c" (UID: "e7cb349e-660d-4ff5-84f7-aa2f64de328c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:19.664157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.664107    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc" pod "e7cb349e-660d-4ff5-84f7-aa2f64de328c" (UID: "e7cb349e-660d-4ff5-84f7-aa2f64de328c"). InnerVolumeSpecName "kube-api-access-dl4wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:19.763681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.763650    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cb349e-660d-4ff5-84f7-aa2f64de328c-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:19.763681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:19.763678    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4wc\" (UniqueName: \"kubernetes.io/projected/e7cb349e-660d-4ff5-84f7-aa2f64de328c-kube-api-access-dl4wc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:20.107651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.107619    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"
Oct 31 01:45:20.107826 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.107618    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc" event={"ID":"c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6","Type":"ContainerDied","Data":"f8b2354799637137f242ee9ab671e1a93f292be390a050bcb964f48515d03cf0"}
Oct 31 01:45:20.107826 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.107780    5419 scope.go:122] "RemoveContainer" containerID="68dd0b9e0c0f0b29cedfd39bb92743015fd4129238b046029972f52fd0a9b762"
Oct 31 01:45:20.109847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.109820    5419 generic.go:356] "Generic (PLEG): container finished" podID="8f2143d0-5d54-4329-a590-c855394930ea" containerID="406a3600820742206b0f99fe56e3a00d8942baf5cfd20dae6ce82cb9936675df" exitCode=60
Oct 31 01:45:20.109990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.109974    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerDied","Data":"406a3600820742206b0f99fe56e3a00d8942baf5cfd20dae6ce82cb9936675df"}
Oct 31 01:45:20.110409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.110391    5419 scope.go:122] "RemoveContainer" containerID="406a3600820742206b0f99fe56e3a00d8942baf5cfd20dae6ce82cb9936675df"
Oct 31 01:45:20.110690 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:20.110667    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv_odh-ai-gateway-infra(8f2143d0-5d54-4329-a590-c855394930ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" podUID="8f2143d0-5d54-4329-a590-c855394930ea"
Oct 31 01:45:20.111495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.111449    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"
Oct 31 01:45:20.111495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.111470    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h" event={"ID":"3c4338a5-329d-43a1-a0ef-b4c7d427b5e7","Type":"ContainerDied","Data":"951b6db04a4c74970d93f4f46b98520e826b27402ef24d092de2849ca244ef2b"}
Oct 31 01:45:20.113077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.113044    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"
Oct 31 01:45:20.113201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.113073    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6" event={"ID":"e7cb349e-660d-4ff5-84f7-aa2f64de328c","Type":"ContainerDied","Data":"4546b6ff35ab54e7bf24c8dec3c1ef083ba4dab88b0a0dad24aaacd335a41450"}
Oct 31 01:45:20.119585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.119569    5419 scope.go:122] "RemoveContainer" containerID="f98e5c4e597e27113b31fb96e52bc1bfbec749052aff3bebe6fc0e2148aa90be"
Oct 31 01:45:20.135470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.135451    5419 scope.go:122] "RemoveContainer" containerID="7f2b23ad28b85bd24aed1af5ad2fecc0e959cb81a1a1deb4a9ad0004c5089354"
Oct 31 01:45:20.145491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.145466    5419 scope.go:122] "RemoveContainer" containerID="763f905c17463d82a8ce81d15959d47f848fdcc75f5fa45afc52ef63d9a2c3bd"
Oct 31 01:45:20.271361 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.271319    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"]
Oct 31 01:45:20.275322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.275294    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890185-2jm6h"]
Oct 31 01:45:20.381846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.381769    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"]
Oct 31 01:45:20.385151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.385112    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-53af08-29890185-k6pdc"]
Oct 31 01:45:20.492234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.492205    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"]
Oct 31 01:45:20.495833 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.495810    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8f6b5b-29890185-4hbz6"]
Oct 31 01:45:20.837591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.837560    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" path="/var/lib/kubelet/pods/3c4338a5-329d-43a1-a0ef-b4c7d427b5e7/volumes"
Oct 31 01:45:20.838015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.838002    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" path="/var/lib/kubelet/pods/c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6/volumes"
Oct 31 01:45:20.838418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:20.838404    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" path="/var/lib/kubelet/pods/e7cb349e-660d-4ff5-84f7-aa2f64de328c/volumes"
Oct 31 01:45:21.614731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.614699    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"]
Oct 31 01:45:21.773631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.773605    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:21.894634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.894546    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca\") pod \"8f2143d0-5d54-4329-a590-c855394930ea\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") "
Oct 31 01:45:21.894634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.894610    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg\" (UniqueName: \"kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg\") pod \"8f2143d0-5d54-4329-a590-c855394930ea\" (UID: \"8f2143d0-5d54-4329-a590-c855394930ea\") "
Oct 31 01:45:21.894963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.894937    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca" pod "8f2143d0-5d54-4329-a590-c855394930ea" (UID: "8f2143d0-5d54-4329-a590-c855394930ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:45:21.895382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.895359    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f2143d0-5d54-4329-a590-c855394930ea-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:21.897366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.897340    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg" pod "8f2143d0-5d54-4329-a590-c855394930ea" (UID: "8f2143d0-5d54-4329-a590-c855394930ea"). InnerVolumeSpecName "kube-api-access-pt7zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:45:21.996735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:21.996697    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7zg\" (UniqueName: \"kubernetes.io/projected/8f2143d0-5d54-4329-a590-c855394930ea-kube-api-access-pt7zg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:45:22.132424 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:22.132399    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"
Oct 31 01:45:22.132598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:22.132396    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv" event={"ID":"8f2143d0-5d54-4329-a590-c855394930ea","Type":"ContainerDied","Data":"24bc2617aad2b7b06f121d04f5cac37d7f3cd1bda31817b59e89756326f3c343"}
Oct 31 01:45:22.132598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:22.132537    5419 scope.go:122] "RemoveContainer" containerID="406a3600820742206b0f99fe56e3a00d8942baf5cfd20dae6ce82cb9936675df"
Oct 31 01:45:22.294163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:22.294116    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"]
Oct 31 01:45:22.297698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:22.297668    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-b2d49b-29890185-rs5jv"]
Oct 31 01:45:22.837009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:22.836967    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2143d0-5d54-4329-a590-c855394930ea" path="/var/lib/kubelet/pods/8f2143d0-5d54-4329-a590-c855394930ea/volumes"
Oct 31 01:45:26.283586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:26.283545    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:45:44.582245 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:45:44.582210    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:45:53.719658 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:45:53.719621    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:46:02.547995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.547956    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"]
Oct 31 01:46:02.548756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548735    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:02.548756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548756    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548775    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548838    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548853    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548859    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548884    5419 cpu_manager.go:414] "containerMap: removing container" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548891    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548905    5419 cpu_manager.go:414] "containerMap: removing container" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548920    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548936    5419 cpu_manager.go:414] "containerMap: removing container" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548943    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.548952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548954    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548961    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548982    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.548989    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549000    5419 cpu_manager.go:414] "containerMap: removing container" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549009    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549019    5419 cpu_manager.go:414] "containerMap: removing container" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549026    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549043    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549049    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549063    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549069    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549115    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549124    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549154    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549161    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549338    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549354    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549364    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549375    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549384    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549394    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549403    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549414    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="76772c3b-0c58-4ddd-a78f-1f91b09b0706" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549426    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549436    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549443    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549453    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549460    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549468    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:02.549667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.549480    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:02.553478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.553459    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593450    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgz2\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593486    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593527    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593612    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593652    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593721    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593742    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593814    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.593858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.593841    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695024    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgz2\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695073    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695119    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695157    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695192    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695233    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695261    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695306    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695347    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.695649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.695521    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.696386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.696106    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.696386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.696106    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.696386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.696273    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.696739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.696450    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.697831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.697808    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.698180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.698160    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.757312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.757282    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"]
Oct 31 01:46:02.760086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.759976    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.760086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.760033    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgz2\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2\") pod \"e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") " pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:02.867321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:02.867240    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:03.297191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:03.297159    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"]
Oct 31 01:46:03.334663 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:46:03.334628    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8c54e6_3039_41c0_af19_75ca5be63d52.slice/crio-a8d24640df4a96a00e3900fbb1a5f097563d148eaecceeb9985e01f3970cd460 WatchSource:0}: Error finding container a8d24640df4a96a00e3900fbb1a5f097563d148eaecceeb9985e01f3970cd460: Status 404 returned error can't find the container with id a8d24640df4a96a00e3900fbb1a5f097563d148eaecceeb9985e01f3970cd460
Oct 31 01:46:03.338077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:03.338034    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:03.338195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:03.338168    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:03.338238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:03.338218    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:03.396388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:03.396352    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" event={"ID":"0a8c54e6-3039-41c0-af19-75ca5be63d52","Type":"ContainerStarted","Data":"a8d24640df4a96a00e3900fbb1a5f097563d148eaecceeb9985e01f3970cd460"}
Oct 31 01:46:04.406834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.406795    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" event={"ID":"0a8c54e6-3039-41c0-af19-75ca5be63d52","Type":"ContainerStarted","Data":"cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90"}
Oct 31 01:46:04.408581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.408556    5419 generic.go:356] "Generic (PLEG): container finished" podID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerID="e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e" exitCode=0
Oct 31 01:46:04.408722 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.408597    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" event={"ID":"0c554ef4-56d2-4e7d-bf35-69c1e5706d56","Type":"ContainerDied","Data":"e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"}
Oct 31 01:46:04.408947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.408935    5419 scope.go:122] "RemoveContainer" containerID="e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"
Oct 31 01:46:04.410360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.410332    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:04.410473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.410416    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:04.410473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.410465    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:04.581399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.581259    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:46:04.614806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.614748    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podStartSLOduration=2.614731372 podStartE2EDuration="2.614731372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:04.573286984 +0000 UTC m=+2485.874243562" watchObservedRunningTime="2026-10-31 01:46:04.614731372 +0000 UTC m=+2485.915687949"
Oct 31 01:46:04.884168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.881334    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:04.917224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.916750    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 01:46:04.917224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:04.916803    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 01:46:05.426232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:05.426189    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" event={"ID":"0c554ef4-56d2-4e7d-bf35-69c1e5706d56","Type":"ContainerStarted","Data":"da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed"}
Oct 31 01:46:05.868317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:05.868287    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 01:46:05.868574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:05.868551    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 01:46:05.923439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:05.923198    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:46:05.927182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:05.927152    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:46:05.927396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:05.927373    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:46:06.868298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:06.868264    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 01:46:06.868718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:06.868328    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 01:46:06.923505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:06.923468    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:46:06.923816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:06.923783    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:46:07.867830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:07.867793    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 01:46:07.868016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:07.867847    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 01:46:07.921164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:07.920417    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:46:07.921164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:07.920486    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:46:08.867542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:08.867506    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 31 01:46:08.867725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:08.867560    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 31 01:46:08.924635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:08.924576    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:46:08.930193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:08.925266    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:46:09.872591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:09.872557    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:09.872973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:09.872952    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:09.873729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:09.873712    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:46:09.924233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:09.924179    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 01:46:09.924444 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:09.924260    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 01:46:10.924550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:10.924522    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:46:10.925050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:10.924995    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:46:10.925709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:10.925690    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:46:14.586410 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:46:14.586334    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:46:23.718941 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:46:23.718912    5419 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@20-35150-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05db315c_10b9_4a39_8656_64ed2a590be1.slice/crio-e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:46:24.576583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:24.576551    5419 generic.go:356] "Generic (PLEG): container finished" podID="05db315c-10b9-4a39-8656-64ed2a590be1" containerID="e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2" exitCode=0
Oct 31 01:46:24.576767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:24.576637    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" event={"ID":"05db315c-10b9-4a39-8656-64ed2a590be1","Type":"ContainerDied","Data":"e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"}
Oct 31 01:46:24.577074 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:24.577060    5419 scope.go:122] "RemoveContainer" containerID="e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"
Oct 31 01:46:24.578575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:24.578549    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:24.578635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:24.578619    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:24.578673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:24.578659    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:25.605966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:25.605933    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" event={"ID":"05db315c-10b9-4a39-8656-64ed2a590be1","Type":"ContainerStarted","Data":"90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce"}
Oct 31 01:46:25.804501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:25.804463    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:46:25.804696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:25.804663    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:46:25.804743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:25.804710    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:46:26.804989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:26.804957    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:46:26.805500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:26.805010    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:46:27.804858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:27.804818    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:46:27.805051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:27.804875    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:46:28.804809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:28.804768    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:46:28.805005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:28.804838    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:46:29.224008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.223967    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"]
Oct 31 01:46:29.224756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224732    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224758    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224785    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224792    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224806    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224813    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cb349e-660d-4ff5-84f7-aa2f64de328c" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224845    5419 cpu_manager.go:414] "containerMap: removing container" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:29.224859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224851    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370a984f-c52e-4f8e-aea3-5667060334a4" containerName="cleanup"
Oct 31 01:46:29.225243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224978    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3afae06-e4c7-4fc5-b6e0-c98cf82aa7e6" containerName="cleanup"
Oct 31 01:46:29.225243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.224999    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4338a5-329d-43a1-a0ef-b4c7d427b5e7" containerName="cleanup"
Oct 31 01:46:29.225243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.225012    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2143d0-5d54-4329-a590-c855394930ea" containerName="cleanup"
Oct 31 01:46:29.228400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.228378    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.328678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.328643    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"]
Oct 31 01:46:29.393162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.384970    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.411667 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.411628    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.411998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.411976    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.412267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.412245    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.412509 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.412489    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.412715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.412697    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.412986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.412965    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.413193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.413171    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.421338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.421309    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qbcf\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.421646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.393798    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e59f12-openshift-default-dockercfg-hcd95\""
Oct 31 01:46:29.527051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.526951    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.527404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.527376    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.527651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.527633    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.527764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.527751    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.527875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.527859    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qbcf\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.527996 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.527981    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.528176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.528157    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.528310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.528295    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.528486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.528470    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.529071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.529050    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.531354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.529478    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.531554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.529694    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.531714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.530291    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.531836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.530495    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.539202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.538764    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.539843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.539820    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.602718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.602676    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.602869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.602798    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qbcf\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf\") pod \"e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") " pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:29.804511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.804411    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 31 01:46:29.804511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.804474    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 31 01:46:29.841884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:29.841852    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:30.299888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.299846    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"]
Oct 31 01:46:30.327175 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:46:30.327126    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc13699e_3ac8_4b5c_a459_d06cbd49c68b.slice/crio-99594a2297c30292ea90e0193ebc47e0503d0ad5b1d19d1bfa3bfb3e3c83e6e7 WatchSource:0}: Error finding container 99594a2297c30292ea90e0193ebc47e0503d0ad5b1d19d1bfa3bfb3e3c83e6e7: Status 404 returned error can't find the container with id 99594a2297c30292ea90e0193ebc47e0503d0ad5b1d19d1bfa3bfb3e3c83e6e7
Oct 31 01:46:30.329691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.329661    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:30.329765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.329730    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:30.329765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.329760    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:30.650916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.650809    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" event={"ID":"fc13699e-3ac8-4b5c-a459-d06cbd49c68b","Type":"ContainerStarted","Data":"3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"}
Oct 31 01:46:30.650916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.650878    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" event={"ID":"fc13699e-3ac8-4b5c-a459-d06cbd49c68b","Type":"ContainerStarted","Data":"99594a2297c30292ea90e0193ebc47e0503d0ad5b1d19d1bfa3bfb3e3c83e6e7"}
Oct 31 01:46:30.769349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.769288    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podStartSLOduration=1.7692728720000002 podStartE2EDuration="1.769272872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:30.767275984 +0000 UTC m=+2512.068232606" watchObservedRunningTime="2026-10-31 01:46:30.769272872 +0000 UTC m=+2512.070229448"
Oct 31 01:46:30.809455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.809423    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:46:30.809907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.809894    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:46:30.810584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.810562    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:46:30.842505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:30.842469    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:31.848808 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:31.848773    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:32.662985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:32.662951    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:32.663985 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:32.663966    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:46:37.367111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.367077    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"]
Oct 31 01:46:37.372000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.371978    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.423997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.423965    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-e59f12\""
Oct 31 01:46:37.424226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.424088    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-e59f12-dockercfg-pkzmc\""
Oct 31 01:46:37.426520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.426482    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"]
Oct 31 01:46:37.430745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.430721    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.430883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.430793    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmhg\" (UniqueName: \"kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg\") pod \"payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.532082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.532020    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.533947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.533923    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmhg\" (UniqueName: \"kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg\") pod \"payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.547757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.532471    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"]
Oct 31 01:46:37.554196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.533736    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.554196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.552008    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.587426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.587394    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"]
Oct 31 01:46:37.608170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.608115    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmhg\" (UniqueName: \"kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg\") pod \"payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.651955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.651888    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.651955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.651928    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x9bz\" (UniqueName: \"kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz\") pod \"payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.686421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.686396    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:37.754037 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.753995    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.754215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.754068    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x9bz\" (UniqueName: \"kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz\") pod \"payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.755216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.755188    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.811554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.811527    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x9bz\" (UniqueName: \"kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz\") pod \"payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:37.868244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:37.868208    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:38.063558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.063529    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"]
Oct 31 01:46:38.065354 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:46:38.065326    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf8ff80_a9b9_4e70_bcbf_5678e966cc7f.slice/crio-238b99501dfde7afbf221b9deabc62ccd7bbdb6daae6725369f3f4ad91524938 WatchSource:0}: Error finding container 238b99501dfde7afbf221b9deabc62ccd7bbdb6daae6725369f3f4ad91524938: Status 404 returned error can't find the container with id 238b99501dfde7afbf221b9deabc62ccd7bbdb6daae6725369f3f4ad91524938
Oct 31 01:46:38.246029 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.246009    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"]
Oct 31 01:46:38.248102 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:46:38.248074    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2c2635_0d1d_49f3_bc35_eefe76a809b6.slice/crio-c9f5d45af343210d08d28fa1539be46677f130a5832d0d4a3374c6d854fbf62e WatchSource:0}: Error finding container c9f5d45af343210d08d28fa1539be46677f130a5832d0d4a3374c6d854fbf62e: Status 404 returned error can't find the container with id c9f5d45af343210d08d28fa1539be46677f130a5832d0d4a3374c6d854fbf62e
Oct 31 01:46:38.702784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.702703    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh" event={"ID":"2d2c2635-0d1d-49f3-bc35-eefe76a809b6","Type":"ContainerStarted","Data":"65939587aacac51a90ac4eb4fbaafc588b518df322cce406c2a7a9143cea05c5"}
Oct 31 01:46:38.702784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.702762    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh" event={"ID":"2d2c2635-0d1d-49f3-bc35-eefe76a809b6","Type":"ContainerStarted","Data":"c9f5d45af343210d08d28fa1539be46677f130a5832d0d4a3374c6d854fbf62e"}
Oct 31 01:46:38.703257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.702791    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:38.704473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.704446    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph" event={"ID":"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f","Type":"ContainerStarted","Data":"15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604"}
Oct 31 01:46:38.704591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.704491    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph" event={"ID":"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f","Type":"ContainerStarted","Data":"238b99501dfde7afbf221b9deabc62ccd7bbdb6daae6725369f3f4ad91524938"}
Oct 31 01:46:38.704591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.704517    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:46:38.817800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:38.817751    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh" podStartSLOduration=1.817735847 podStartE2EDuration="1.817735847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:38.815488428 +0000 UTC m=+2520.116445006" watchObservedRunningTime="2026-10-31 01:46:38.817735847 +0000 UTC m=+2520.118692475"
Oct 31 01:46:40.179561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.179511    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph" podStartSLOduration=3.179497637 podStartE2EDuration="3.179497637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:38.945605375 +0000 UTC m=+2520.246561952" watchObservedRunningTime="2026-10-31 01:46:40.179497637 +0000 UTC m=+2521.480454214"
Oct 31 01:46:40.180708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.180690    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"]
Oct 31 01:46:40.185269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.185249    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.236952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.236913    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-99145d-openshift-default-dockercfg-g7kns\""
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286293    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286483    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286522    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286569    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286612    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvdzg\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286677    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286769    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286805    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.286913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.286870    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.387607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.387554    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvdzg\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.387913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.387653    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.387913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.387726    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.387913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.387757    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.387913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.387815    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388054    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388153    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388183    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388231    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388298    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388812    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.388838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.388826    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.389618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.389575    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.389750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.389620    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.391442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.391415    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.391680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.391661    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.392911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.392889    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"]
Oct 31 01:46:40.449428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.449355    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvdzg\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.449567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.449478    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token\") pod \"e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") " pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.498835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.498799    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:40.926165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.926071    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"]
Oct 31 01:46:40.927523 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:46:40.927498    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccffd2b0_3dcb_42b3_adb1_3c57482d6ea8.slice/crio-e13fbaf0d5f91a1f9924ab918b1a9f5255e4a3bfd093e1c9ddd3e093d83c7bf2 WatchSource:0}: Error finding container e13fbaf0d5f91a1f9924ab918b1a9f5255e4a3bfd093e1c9ddd3e093d83c7bf2: Status 404 returned error can't find the container with id e13fbaf0d5f91a1f9924ab918b1a9f5255e4a3bfd093e1c9ddd3e093d83c7bf2
Oct 31 01:46:40.930290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.930258    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:40.930373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.930324    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:40.930373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:40.930352    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:46:41.730187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:41.730125    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" event={"ID":"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8","Type":"ContainerStarted","Data":"5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"}
Oct 31 01:46:41.730629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:41.730199    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" event={"ID":"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8","Type":"ContainerStarted","Data":"e13fbaf0d5f91a1f9924ab918b1a9f5255e4a3bfd093e1c9ddd3e093d83c7bf2"}
Oct 31 01:46:41.897176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:41.897101    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podStartSLOduration=1.897088442 podStartE2EDuration="1.897088442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:41.895571839 +0000 UTC m=+2523.196528416" watchObservedRunningTime="2026-10-31 01:46:41.897088442 +0000 UTC m=+2523.198045085"
Oct 31 01:46:42.499938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:42.499903    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:42.506255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:42.506230    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:42.736543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:42.736514    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:42.737596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:42.737575    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:46:43.933922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:43.933892    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:46:44.582064 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:46:44.582025    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:46:49.718655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:49.718573    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:46:49.718655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:46:49.718649    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:47:14.582560 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:47:14.582514    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:47:17.359892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.359852    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"]
Oct 31 01:47:17.365268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.365250    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.403822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.403791    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.403822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.403829    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.403978    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svv7t\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.404021    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.404177    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.404219    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.404242    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.404273    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.404446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.404300    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.418485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.418459    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cd0c98-openshift-default-dockercfg-xf86x\""
Oct 31 01:47:17.469784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.469748    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"]
Oct 31 01:47:17.505454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.505402    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.506398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.506372    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.506986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.506956    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.508476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.508449    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svv7t\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.509210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.509188    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.509779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.509760    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.510041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.509999    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.510273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.510253    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.510501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.510484    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.510685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.510664    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.511207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.511181    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.511359 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.509614    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.514311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.514287    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.538695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.523792    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.546846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.546791    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.557685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.557643    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.600166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.592179    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svv7t\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.600166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.592642    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:17.680294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:17.680224    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:18.115477 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:47:18.115447    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dc475d_a4c7_4510_9ebd_98d03c4adbe9.slice/crio-8506f81162e5158dfa2576812cfd6fadef9436975412bbe7c145dca91d81fc80 WatchSource:0}: Error finding container 8506f81162e5158dfa2576812cfd6fadef9436975412bbe7c145dca91d81fc80: Status 404 returned error can't find the container with id 8506f81162e5158dfa2576812cfd6fadef9436975412bbe7c145dca91d81fc80
Oct 31 01:47:18.118065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:18.118026    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:47:18.118198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:18.118181    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:47:18.118280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:18.118240    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:47:18.153238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:18.153189    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"]
Oct 31 01:47:18.992306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:18.992268    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" event={"ID":"51dc475d-a4c7-4510-9ebd-98d03c4adbe9","Type":"ContainerStarted","Data":"afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"}
Oct 31 01:47:18.992701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:18.992329    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" event={"ID":"51dc475d-a4c7-4510-9ebd-98d03c4adbe9","Type":"ContainerStarted","Data":"8506f81162e5158dfa2576812cfd6fadef9436975412bbe7c145dca91d81fc80"}
Oct 31 01:47:19.018350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:19.018318    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:47:19.110521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:19.110470    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podStartSLOduration=2.110456008 podStartE2EDuration="2.110456008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:19.109131415 +0000 UTC m=+2560.410087993" watchObservedRunningTime="2026-10-31 01:47:19.110456008 +0000 UTC m=+2560.411412585"
Oct 31 01:47:19.680785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:19.680747    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:19.686618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:19.686594    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:20.028162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:20.027351    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:20.035158 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:20.029553    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:47:28.583222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:28.583189    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx_765496b6-57f5-419b-bec5-93a7758586f3/payload-processing/0.log"
Oct 31 01:47:29.400485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:29.400455    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm_d3da6392-a65c-483e-ae2b-35ad51a4b4db/payload-processing/0.log"
Oct 31 01:47:29.623848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:29.623817    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm_d3da6392-a65c-483e-ae2b-35ad51a4b4db/payload-processing/0.log"
Oct 31 01:47:30.522185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.522158    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"]
Oct 31 01:47:30.526363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.526346    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.563273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.563240    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.563426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.563286    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.563426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.563379    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxt6m\" (UniqueName: \"kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.582089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.582063    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"]
Oct 31 01:47:30.628226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.628200    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:47:30.628226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.628200    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:47:30.628226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.628200    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:47:30.628680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.628242    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:47:30.664301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.664276    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.664456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.664329    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.664456 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.664409    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxt6m\" (UniqueName: \"kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.665150 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.665074    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.667038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.667011    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.720504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.720475    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxt6m\" (UniqueName: \"kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m\") pod \"maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:30.840690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:30.840616    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:31.218221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:31.218193    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"]
Oct 31 01:47:31.219947 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:47:31.219922    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d173dd6_cd28_4cd3_a319_9900221763af.slice/crio-23b9b5489e1f4093a5060b6d4427f8b3a32269ca9bfa24ad48ba9b1f022d2142 WatchSource:0}: Error finding container 23b9b5489e1f4093a5060b6d4427f8b3a32269ca9bfa24ad48ba9b1f022d2142: Status 404 returned error can't find the container with id 23b9b5489e1f4093a5060b6d4427f8b3a32269ca9bfa24ad48ba9b1f022d2142
Oct 31 01:47:32.106917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:32.106874    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74" event={"ID":"9d173dd6-cd28-4cd3-a319-9900221763af","Type":"ContainerStarted","Data":"fb7122dddfee1494ce7ea250912784c853b1081041f1951d9e167146f57d55c2"}
Oct 31 01:47:32.107486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:32.106982    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74" event={"ID":"9d173dd6-cd28-4cd3-a319-9900221763af","Type":"ContainerStarted","Data":"23b9b5489e1f4093a5060b6d4427f8b3a32269ca9bfa24ad48ba9b1f022d2142"}
Oct 31 01:47:32.221856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:32.219447    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74" podStartSLOduration=2.219430051 podStartE2EDuration="2.219430051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:32.218749211 +0000 UTC m=+2573.519705810" watchObservedRunningTime="2026-10-31 01:47:32.219430051 +0000 UTC m=+2573.520386629"
Oct 31 01:47:33.115904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:33.115868    5419 generic.go:356] "Generic (PLEG): container finished" podID="9d173dd6-cd28-4cd3-a319-9900221763af" containerID="fb7122dddfee1494ce7ea250912784c853b1081041f1951d9e167146f57d55c2" exitCode=0
Oct 31 01:47:33.116360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:33.115938    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74" event={"ID":"9d173dd6-cd28-4cd3-a319-9900221763af","Type":"ContainerDied","Data":"fb7122dddfee1494ce7ea250912784c853b1081041f1951d9e167146f57d55c2"}
Oct 31 01:47:34.279480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.279456    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:34.411543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.411462    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca\") pod \"9d173dd6-cd28-4cd3-a319-9900221763af\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") "
Oct 31 01:47:34.411745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.411560    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m\" (UniqueName: \"kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m\") pod \"9d173dd6-cd28-4cd3-a319-9900221763af\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") "
Oct 31 01:47:34.411745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.411595    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca\") pod \"9d173dd6-cd28-4cd3-a319-9900221763af\" (UID: \"9d173dd6-cd28-4cd3-a319-9900221763af\") "
Oct 31 01:47:34.411914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.411883    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca" pod "9d173dd6-cd28-4cd3-a319-9900221763af" (UID: "9d173dd6-cd28-4cd3-a319-9900221763af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:47:34.413879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.413843    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m" pod "9d173dd6-cd28-4cd3-a319-9900221763af" (UID: "9d173dd6-cd28-4cd3-a319-9900221763af"). InnerVolumeSpecName "kube-api-access-xxt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:47:34.414014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.413877    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca" pod "9d173dd6-cd28-4cd3-a319-9900221763af" (UID: "9d173dd6-cd28-4cd3-a319-9900221763af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:47:34.513463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.513432    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:47:34.513463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.513458    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxt6m\" (UniqueName: \"kubernetes.io/projected/9d173dd6-cd28-4cd3-a319-9900221763af-kube-api-access-xxt6m\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:47:34.513463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:34.513470    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d173dd6-cd28-4cd3-a319-9900221763af-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:47:35.130205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:35.130117    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"
Oct 31 01:47:35.130205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:35.130166    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74" event={"ID":"9d173dd6-cd28-4cd3-a319-9900221763af","Type":"ContainerDied","Data":"23b9b5489e1f4093a5060b6d4427f8b3a32269ca9bfa24ad48ba9b1f022d2142"}
Oct 31 01:47:35.130400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:35.130216    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b9b5489e1f4093a5060b6d4427f8b3a32269ca9bfa24ad48ba9b1f022d2142"
Oct 31 01:47:40.650030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:40.650000    5419 scope.go:122] "RemoveContainer" containerID="807281cac5628ace84b28d3489f5d3f2f6b6cf5e4b3f9f349a46d16ac127cc8e"
Oct 31 01:47:44.582186 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:47:44.582129    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:47:49.734763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.734727    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"]
Oct 31 01:47:49.735450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.735433    5419 cpu_manager.go:414] "containerMap: removing container" podUID="9d173dd6-cd28-4cd3-a319-9900221763af" containerName="revoke-keys"
Oct 31 01:47:49.735519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.735452    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d173dd6-cd28-4cd3-a319-9900221763af" containerName="revoke-keys"
Oct 31 01:47:49.735558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.735553    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d173dd6-cd28-4cd3-a319-9900221763af" containerName="revoke-keys"
Oct 31 01:47:49.740476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.740460    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.835762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.835737    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2c0de9-openshift-default-dockercfg-p54mv\""
Oct 31 01:47:49.845564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.845530    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"]
Oct 31 01:47:49.896354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.896303    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.896824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.896799    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.897077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.897056    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2fqf\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.897294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.897265    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.897505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.897487    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.897713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.897675    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.897914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.897898    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.898118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.898101    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.898280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.898263    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999305 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999218    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999337    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999396    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2fqf\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999448    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999489    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999521    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999554    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999610    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999644    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:49.999734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:49.999676    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.000411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.000382    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.000572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.000387    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.000735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.000699    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.000735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.000717    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.003543 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.003521    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.003741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.003724    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.056773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.056738    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.057035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.057015    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2fqf\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf\") pod \"e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") " pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.354715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.354627    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:50.788617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.788593    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"]
Oct 31 01:47:50.790003 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:47:50.789972    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fdbaaf_1f43_4baf_a00d_4fe6d9e1fb97.slice/crio-b69ffd622c4b22cffefc7ecbf2eced831954e1b1d82a4dad2a7db9005928917d WatchSource:0}: Error finding container b69ffd622c4b22cffefc7ecbf2eced831954e1b1d82a4dad2a7db9005928917d: Status 404 returned error can't find the container with id b69ffd622c4b22cffefc7ecbf2eced831954e1b1d82a4dad2a7db9005928917d
Oct 31 01:47:50.792625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.792595    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:47:50.792704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.792661    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:47:50.792704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:50.792692    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:47:51.239397 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:51.239363    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" event={"ID":"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97","Type":"ContainerStarted","Data":"4ff5f0d5a424e6a567972a20e9ce95d212ca78c2cb589557587025fa9644933e"}
Oct 31 01:47:51.239589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:51.239418    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" event={"ID":"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97","Type":"ContainerStarted","Data":"b69ffd622c4b22cffefc7ecbf2eced831954e1b1d82a4dad2a7db9005928917d"}
Oct 31 01:47:51.355303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:51.355272    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:51.356483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:51.356430    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podStartSLOduration=2.3564144750000002 podStartE2EDuration="2.356414475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:51.355664133 +0000 UTC m=+2592.656620714" watchObservedRunningTime="2026-10-31 01:47:51.356414475 +0000 UTC m=+2592.657371053"
Oct 31 01:47:51.362035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:51.362014    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:52.245945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:52.245913    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:47:52.246967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:47:52.246943    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:48:08.055457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:08.055417    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:48:14.584111 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:48:14.584063    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:48:18.100686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.100003    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": read tcp 10.133.0.2:53492->10.133.0.210:15021: read: connection reset by peer" start-of-body=
Oct 31 01:48:18.100686 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.100066    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": read tcp 10.133.0.2:53492->10.133.0.210:15021: read: connection reset by peer"
Oct 31 01:48:18.457400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.457294    5419 generic.go:356] "Generic (PLEG): container finished" podID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerID="3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139" exitCode=0
Oct 31 01:48:18.457400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.457362    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" event={"ID":"fc13699e-3ac8-4b5c-a459-d06cbd49c68b","Type":"ContainerDied","Data":"3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"}
Oct 31 01:48:18.457861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.457844    5419 scope.go:122] "RemoveContainer" containerID="3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"
Oct 31 01:48:18.459898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.459867    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:48:18.459999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.459943    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:48:18.459999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:18.459977    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:48:19.470665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:19.470630    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" event={"ID":"fc13699e-3ac8-4b5c-a459-d06cbd49c68b","Type":"ContainerStarted","Data":"f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822"}
Oct 31 01:48:19.842233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:19.842123    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:48:19.842395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:19.842350    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 01:48:19.842439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:19.842388    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 01:48:20.842419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:20.842387    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 01:48:20.842835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:20.842436    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 01:48:21.842684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:21.842650    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 01:48:21.843091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:21.842707    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 01:48:22.843197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:22.843167    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 01:48:22.843575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:22.843218    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 01:48:23.843093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:23.843057    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 01:48:23.843300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:23.843107    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 01:48:24.913008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:24.912970    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:48:24.914164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:24.914132    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:48:24.915828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:24.915262    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:48:36.750381 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:36.750337    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:48:39.048472 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:39.048438    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"]
Oct 31 01:48:39.100889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:39.100851    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6141e20-kntqw"]
Oct 31 01:48:40.702586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:40.702551    5419 scope.go:122] "RemoveContainer" containerID="5faa4fca6978ac35c3188e9c1fbc4a2e9bdd7d8e4f4a6f2b6f715773f265c45c"
Oct 31 01:48:40.839944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:40.839913    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682742b0-56a9-4fdd-b341-f83b60f41c75" path="/var/lib/kubelet/pods/682742b0-56a9-4fdd-b341-f83b60f41c75/volumes"
Oct 31 01:48:44.587205 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:48:44.587132    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:48:47.690117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:47.690083    5419 generic.go:356] "Generic (PLEG): container finished" podID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerID="5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d" exitCode=0
Oct 31 01:48:47.690610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:47.690181    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" event={"ID":"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8","Type":"ContainerDied","Data":"5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"}
Oct 31 01:48:47.690610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:47.690578    5419 scope.go:122] "RemoveContainer" containerID="5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"
Oct 31 01:48:47.695918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:47.695885    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:48:47.696034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:47.695972    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:48:47.696034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:47.696017    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:48:48.710888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:48.710854    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" event={"ID":"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8","Type":"ContainerStarted","Data":"accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606"}
Oct 31 01:48:49.047380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:49.047352    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"]
Oct 31 01:48:49.049762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:49.049742    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9198562-nnl6p"]
Oct 31 01:48:49.499366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:49.499325    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:48:49.499701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:49.499671    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:48:49.500040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:49.500009    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:48:50.499321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:50.499279    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:48:50.499749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:50.499345    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:48:50.838188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:50.838101    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f956485-7ab2-4987-81af-4723748e67fb" path="/var/lib/kubelet/pods/1f956485-7ab2-4987-81af-4723748e67fb/volumes"
Oct 31 01:48:51.499629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:51.499589    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:48:51.500200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:51.499651    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:48:52.499601 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:52.499556    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:48:52.499776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:52.499630    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:48:53.499623 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:53.499573    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 01:48:53.499894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:53.499664    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 01:48:54.504094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:54.504063    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:48:54.504612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:54.504591    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:48:54.505274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:48:54.505256    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:49:04.158675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.158643    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"]
Oct 31 01:49:04.166312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.166277    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.220302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.220273    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"]
Oct 31 01:49:04.265956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.265923    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:49:04.266117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.265962    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:49:04.266117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.265980    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:49:04.267771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.267759    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:49:04.282401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.282376    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.282525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.282472    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcvnb\" (UniqueName: \"kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.282605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.282588    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.384243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.384212    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.384395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.384262    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcvnb\" (UniqueName: \"kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.384395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.384334    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.385016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.384996    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.386675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.386653    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.440880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.440816    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcvnb\" (UniqueName: \"kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb\") pod \"maas-api-revoke-sub-7ba569dd-fqw2t\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.481755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.481728    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:04.856623 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:04.856590    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"]
Oct 31 01:49:04.857866 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:49:04.857843    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ab0ad8_32ba_4b81_8ec9_6ba7a48d968e.slice/crio-a6be588d60f5ef3bbeeaac5bd8a6e53c97e2eb4fba8a9f63a2c79fcf1ac290e5 WatchSource:0}: Error finding container a6be588d60f5ef3bbeeaac5bd8a6e53c97e2eb4fba8a9f63a2c79fcf1ac290e5: Status 404 returned error can't find the container with id a6be588d60f5ef3bbeeaac5bd8a6e53c97e2eb4fba8a9f63a2c79fcf1ac290e5
Oct 31 01:49:05.820394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:05.820362    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t" event={"ID":"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e","Type":"ContainerStarted","Data":"b86259845834a2facbd7dfee2b1da5fc58f27b3c35c6ccdbe50a6cdbe720d7c1"}
Oct 31 01:49:05.820788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:05.820421    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t" event={"ID":"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e","Type":"ContainerStarted","Data":"a6be588d60f5ef3bbeeaac5bd8a6e53c97e2eb4fba8a9f63a2c79fcf1ac290e5"}
Oct 31 01:49:05.938211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:05.938154    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t" podStartSLOduration=1.9381301039999999 podStartE2EDuration="1.938130104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:05.935091477 +0000 UTC m=+2667.236048053" watchObservedRunningTime="2026-10-31 01:49:05.938130104 +0000 UTC m=+2667.239086718"
Oct 31 01:49:06.828506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:06.828472    5419 generic.go:356] "Generic (PLEG): container finished" podID="04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" containerID="b86259845834a2facbd7dfee2b1da5fc58f27b3c35c6ccdbe50a6cdbe720d7c1" exitCode=0
Oct 31 01:49:06.828908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:06.828556    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t" event={"ID":"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e","Type":"ContainerDied","Data":"b86259845834a2facbd7dfee2b1da5fc58f27b3c35c6ccdbe50a6cdbe720d7c1"}
Oct 31 01:49:07.982390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:07.982362    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:08.129242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.129162    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca\") pod \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") "
Oct 31 01:49:08.129242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.129200    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca\") pod \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") "
Oct 31 01:49:08.129450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.129272    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb\" (UniqueName: \"kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb\") pod \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\" (UID: \"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e\") "
Oct 31 01:49:08.129577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.129553    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca" pod "04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" (UID: "04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:08.130144 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.130065    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:08.131680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.131650    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca" pod "04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" (UID: "04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:08.131773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.131669    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb" pod "04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" (UID: "04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e"). InnerVolumeSpecName "kube-api-access-rcvnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:08.231498 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.231464    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:08.231498 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.231494    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcvnb\" (UniqueName: \"kubernetes.io/projected/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e-kube-api-access-rcvnb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:08.849656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.849619    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"
Oct 31 01:49:08.849842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.849617    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t" event={"ID":"04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e","Type":"ContainerDied","Data":"a6be588d60f5ef3bbeeaac5bd8a6e53c97e2eb4fba8a9f63a2c79fcf1ac290e5"}
Oct 31 01:49:08.849842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:08.849753    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6be588d60f5ef3bbeeaac5bd8a6e53c97e2eb4fba8a9f63a2c79fcf1ac290e5"
Oct 31 01:49:10.324264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:10.324230    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:49:14.606556 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:49:14.606510    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:49:18.403568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.403530    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"]
Oct 31 01:49:18.404175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.404159    5419 cpu_manager.go:414] "containerMap: removing container" podUID="04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" containerName="revoke-keys"
Oct 31 01:49:18.404175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.404176    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" containerName="revoke-keys"
Oct 31 01:49:18.404364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.404354    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" containerName="revoke-keys"
Oct 31 01:49:18.408400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.408380    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.479950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.479909    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480034    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480157    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85x5\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480240    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480373    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480415    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480440    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480556    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.480714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.480586    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.581657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.581620    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.581861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.581691    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.581861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.581774    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85x5\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.581861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.581801    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.582042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.581890    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.582106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.582046    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.582202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.582096    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.582306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.581925    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.582380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.582351    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.582946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.582922    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.583067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.583020    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.583067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.583051    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.583399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.583377    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.583484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.583437    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.584940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.584916    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.585044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.585015    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.617405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.617368    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"]
Oct 31 01:49:18.639014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.638978    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.639243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.638991    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85x5\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5\") pod \"e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") " pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:18.722621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:18.722584    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:19.162422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.162392    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"]
Oct 31 01:49:19.163640 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:49:19.163606    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode945423f_962d_4713_9925_08bc6f546dbe.slice/crio-fa395fa29c87acc1809b29c4edb9ca31b53ba4bc9bf2d9cc5e72d1b74d51c4d5 WatchSource:0}: Error finding container fa395fa29c87acc1809b29c4edb9ca31b53ba4bc9bf2d9cc5e72d1b74d51c4d5: Status 404 returned error can't find the container with id fa395fa29c87acc1809b29c4edb9ca31b53ba4bc9bf2d9cc5e72d1b74d51c4d5
Oct 31 01:49:19.166974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.166938    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:19.167091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.167049    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:19.167185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.167097    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:19.927348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.927315    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"]
Oct 31 01:49:19.932692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.932666    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:19.958603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.958574    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" event={"ID":"e945423f-962d-4713-9925-08bc6f546dbe","Type":"ContainerStarted","Data":"2699c7dc2540be3c9cc85cc469a55d16bcc75b2f07348be6f445bbb076aadc67"}
Oct 31 01:49:19.958714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:19.958627    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" event={"ID":"e945423f-962d-4713-9925-08bc6f546dbe","Type":"ContainerStarted","Data":"fa395fa29c87acc1809b29c4edb9ca31b53ba4bc9bf2d9cc5e72d1b74d51c4d5"}
Oct 31 01:49:20.010469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.010429    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.010802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.010780    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgswh\" (UniqueName: \"kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.011040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.011018    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.011327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.010595    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:49:20.011327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.010510    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:49:20.011327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.011069    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"]
Oct 31 01:49:20.031879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.031845    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:49:20.031997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.031951    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:49:20.112888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.112857    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.113101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.112920    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgswh\" (UniqueName: \"kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.113101 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.113006    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.113596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.113569    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.115500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.115480    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.167627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.167562    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podStartSLOduration=2.167547031 podStartE2EDuration="2.167547031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:20.165526906 +0000 UTC m=+2681.466483485" watchObservedRunningTime="2026-10-31 01:49:20.167547031 +0000 UTC m=+2681.468503608"
Oct 31 01:49:20.170298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.170270    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgswh\" (UniqueName: \"kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh\") pod \"maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.247020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.246985    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:20.660832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.660801    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"]
Oct 31 01:49:20.663053 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:49:20.663022    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43872e4_7f61_4d55_a513_64d2c542e378.slice/crio-4109465d6281d542c1e601017c2770059cb9267bc16996b08bafe2a04e197083 WatchSource:0}: Error finding container 4109465d6281d542c1e601017c2770059cb9267bc16996b08bafe2a04e197083: Status 404 returned error can't find the container with id 4109465d6281d542c1e601017c2770059cb9267bc16996b08bafe2a04e197083
Oct 31 01:49:20.723133 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.722863    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:20.724830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.724800    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 31 01:49:20.724969 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.724896    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 31 01:49:20.966335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:20.966289    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj" event={"ID":"a43872e4-7f61-4d55-a513-64d2c542e378","Type":"ContainerStarted","Data":"4109465d6281d542c1e601017c2770059cb9267bc16996b08bafe2a04e197083"}
Oct 31 01:49:21.070271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.070237    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:49:21.070455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.070294    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:49:21.723503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.723476    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 31 01:49:21.723760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.723523    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 31 01:49:22.001300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.974167    5419 generic.go:356] "Generic (PLEG): container finished" podID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerID="afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30" exitCode=0
Oct 31 01:49:22.001300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.974194    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" event={"ID":"51dc475d-a4c7-4510-9ebd-98d03c4adbe9","Type":"ContainerDied","Data":"afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"}
Oct 31 01:49:22.001300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:21.974690    5419 scope.go:122] "RemoveContainer" containerID="afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"
Oct 31 01:49:22.003715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.003676    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:22.003854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.003755    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:22.003854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.003796    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:22.005357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.005329    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj" event={"ID":"a43872e4-7f61-4d55-a513-64d2c542e378","Type":"ContainerDied","Data":"49c61d798e78fc6226fd7dd62e6414c911af1f6e6fb28b96147de1895a8dfe1f"}
Oct 31 01:49:22.005654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.005632    5419 generic.go:356] "Generic (PLEG): container finished" podID="a43872e4-7f61-4d55-a513-64d2c542e378" containerID="49c61d798e78fc6226fd7dd62e6414c911af1f6e6fb28b96147de1895a8dfe1f" exitCode=0
Oct 31 01:49:22.723702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.723648    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 31 01:49:22.723945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:22.723711    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 31 01:49:23.035834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.035742    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" event={"ID":"51dc475d-a4c7-4510-9ebd-98d03c4adbe9","Type":"ContainerStarted","Data":"62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b"}
Oct 31 01:49:23.338473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.338408    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:23.355027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.354999    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca\") pod \"a43872e4-7f61-4d55-a513-64d2c542e378\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") "
Oct 31 01:49:23.355197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.355074    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh\" (UniqueName: \"kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh\") pod \"a43872e4-7f61-4d55-a513-64d2c542e378\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") "
Oct 31 01:49:23.355245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.355198    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca\") pod \"a43872e4-7f61-4d55-a513-64d2c542e378\" (UID: \"a43872e4-7f61-4d55-a513-64d2c542e378\") "
Oct 31 01:49:23.355621 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.355579    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca" pod "a43872e4-7f61-4d55-a513-64d2c542e378" (UID: "a43872e4-7f61-4d55-a513-64d2c542e378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:23.357516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.357493    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh" pod "a43872e4-7f61-4d55-a513-64d2c542e378" (UID: "a43872e4-7f61-4d55-a513-64d2c542e378"). InnerVolumeSpecName "kube-api-access-qgswh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:23.357733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.357708    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca" pod "a43872e4-7f61-4d55-a513-64d2c542e378" (UID: "a43872e4-7f61-4d55-a513-64d2c542e378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:23.456534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.456493    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:23.456534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.456522    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgswh\" (UniqueName: \"kubernetes.io/projected/a43872e4-7f61-4d55-a513-64d2c542e378-kube-api-access-qgswh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:23.456534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.456533    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43872e4-7f61-4d55-a513-64d2c542e378-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:23.696223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.683555    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:49:23.696223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.683696    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 31 01:49:23.696223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.683754    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 31 01:49:23.724669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.724595    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 31 01:49:23.724669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:23.724655    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 31 01:49:24.045899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.045871    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"
Oct 31 01:49:24.046291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.045902    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj" event={"ID":"a43872e4-7f61-4d55-a513-64d2c542e378","Type":"ContainerDied","Data":"4109465d6281d542c1e601017c2770059cb9267bc16996b08bafe2a04e197083"}
Oct 31 01:49:24.046291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.045952    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4109465d6281d542c1e601017c2770059cb9267bc16996b08bafe2a04e197083"
Oct 31 01:49:24.690565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.690523    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 31 01:49:24.690859 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.690828    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 31 01:49:24.723089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.722964    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body=
Oct 31 01:49:24.723089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:24.723023    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused"
Oct 31 01:49:25.691165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:25.690535    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 31 01:49:25.691165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:25.690621    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 31 01:49:25.728229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:25.728198    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:25.728507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:25.728473    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:25.729281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:25.729265    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:49:26.689588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:26.689545    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 31 01:49:26.689895 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:26.689863    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 31 01:49:27.687582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:27.687544    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body=
Oct 31 01:49:27.688244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:27.688206    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused"
Oct 31 01:49:28.685708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:28.685676    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:49:28.686088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:28.686062    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:49:28.686669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:28.686649    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:49:31.451767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:31.451735    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"]
Oct 31 01:49:31.556047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:31.555998    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"]
Oct 31 01:49:36.452330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.452284    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9" podUID="578c5687-d459-454e-b2c2-5567165e8945" containerName="payload-processing" containerID="cri-o://9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c" gracePeriod=25
Oct 31 01:49:36.556767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.556723    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5" podUID="644432a3-30e0-42eb-bf49-f53cc9a4f1b5" containerName="payload-pre-processing" containerID="cri-o://0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c" gracePeriod=25
Oct 31 01:49:36.769913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.769886    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:49:36.906165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.906040    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume\") pod \"578c5687-d459-454e-b2c2-5567165e8945\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") "
Oct 31 01:49:36.906165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.906157    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr\" (UniqueName: \"kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr\") pod \"578c5687-d459-454e-b2c2-5567165e8945\" (UID: \"578c5687-d459-454e-b2c2-5567165e8945\") "
Oct 31 01:49:36.907031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.907001    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume" pod "578c5687-d459-454e-b2c2-5567165e8945" (UID: "578c5687-d459-454e-b2c2-5567165e8945"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:36.909838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.909794    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr" pod "578c5687-d459-454e-b2c2-5567165e8945" (UID: "578c5687-d459-454e-b2c2-5567165e8945"). InnerVolumeSpecName "kube-api-access-gs7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:36.936810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:36.936789    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:49:37.007485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.007449    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578c5687-d459-454e-b2c2-5567165e8945-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:37.007485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.007486    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7qr\" (UniqueName: \"kubernetes.io/projected/578c5687-d459-454e-b2c2-5567165e8945-kube-api-access-gs7qr\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:37.108049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.108013    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume\") pod \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") "
Oct 31 01:49:37.108310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.108243    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh\" (UniqueName: \"kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh\") pod \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\" (UID: \"644432a3-30e0-42eb-bf49-f53cc9a4f1b5\") "
Oct 31 01:49:37.108532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.108504    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume" pod "644432a3-30e0-42eb-bf49-f53cc9a4f1b5" (UID: "644432a3-30e0-42eb-bf49-f53cc9a4f1b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:37.108787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.108756    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:37.110645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.110621    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh" pod "644432a3-30e0-42eb-bf49-f53cc9a4f1b5" (UID: "644432a3-30e0-42eb-bf49-f53cc9a4f1b5"). InnerVolumeSpecName "kube-api-access-l2wdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:37.148257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.148226    5419 generic.go:356] "Generic (PLEG): container finished" podID="578c5687-d459-454e-b2c2-5567165e8945" containerID="9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c" exitCode=0
Oct 31 01:49:37.148416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.148277    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"
Oct 31 01:49:37.148416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.148311    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9" event={"ID":"578c5687-d459-454e-b2c2-5567165e8945","Type":"ContainerDied","Data":"9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c"}
Oct 31 01:49:37.148416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.148364    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9" event={"ID":"578c5687-d459-454e-b2c2-5567165e8945","Type":"ContainerDied","Data":"717e33fd2d895938e21cdb9ecd76d0c8b315d6d62728de3c3c19796ddcddaf10"}
Oct 31 01:49:37.148416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.148391    5419 scope.go:122] "RemoveContainer" containerID="9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c"
Oct 31 01:49:37.149877 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.149849    5419 generic.go:356] "Generic (PLEG): container finished" podID="644432a3-30e0-42eb-bf49-f53cc9a4f1b5" containerID="0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c" exitCode=0
Oct 31 01:49:37.150186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.149905    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"
Oct 31 01:49:37.150186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.149930    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5" event={"ID":"644432a3-30e0-42eb-bf49-f53cc9a4f1b5","Type":"ContainerDied","Data":"0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c"}
Oct 31 01:49:37.150186 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.149976    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5" event={"ID":"644432a3-30e0-42eb-bf49-f53cc9a4f1b5","Type":"ContainerDied","Data":"d37af6196688f48e9bd7990905045fff43f0f9cfd832fe2189d7e73376d7e2bb"}
Oct 31 01:49:37.164527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.159630    5419 scope.go:122] "RemoveContainer" containerID="9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c"
Oct 31 01:49:37.164527 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:49:37.159973    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c\": container with ID starting with 9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c not found: ID does not exist" containerID="9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c"
Oct 31 01:49:37.164527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.159998    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c"} err="failed to get container status \"9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c\": rpc error: code = NotFound desc = could not find container \"9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c\": container with ID starting with 9734d4b174e7251a817e8098bf57cf4cb78359987b55035d3ec6fa5e763ebc5c not found: ID does not exist"
Oct 31 01:49:37.164527 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.160014    5419 scope.go:122] "RemoveContainer" containerID="0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c"
Oct 31 01:49:37.172288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.172270    5419 scope.go:122] "RemoveContainer" containerID="0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c"
Oct 31 01:49:37.172556 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:49:37.172537    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c\": container with ID starting with 0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c not found: ID does not exist" containerID="0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c"
Oct 31 01:49:37.172613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.172563    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c"} err="failed to get container status \"0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c\": rpc error: code = NotFound desc = could not find container \"0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c\": container with ID starting with 0eacfa589a762188689d400ddac93931f1013190d005f5c3c2c394736fa4758c not found: ID does not exist"
Oct 31 01:49:37.210531 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.210498    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2wdh\" (UniqueName: \"kubernetes.io/projected/644432a3-30e0-42eb-bf49-f53cc9a4f1b5-kube-api-access-l2wdh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:37.311434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.311354    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"]
Oct 31 01:49:37.313878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.313856    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8a3796-5954d5d847-c4bm9"]
Oct 31 01:49:37.422102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.422071    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"]
Oct 31 01:49:37.424680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:37.424657    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8a3796-df76b56d7-bjwb5"]
Oct 31 01:49:38.838290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:38.838253    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578c5687-d459-454e-b2c2-5567165e8945" path="/var/lib/kubelet/pods/578c5687-d459-454e-b2c2-5567165e8945/volumes"
Oct 31 01:49:38.838817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:38.838779    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644432a3-30e0-42eb-bf49-f53cc9a4f1b5" path="/var/lib/kubelet/pods/644432a3-30e0-42eb-bf49-f53cc9a4f1b5/volumes"
Oct 31 01:49:40.740312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:40.740233    5419 scope.go:122] "RemoveContainer" containerID="1e8bf4bb2a9f7593cda572fec5e0494ac37be3d59a4827ce9e812197c61e42a0"
Oct 31 01:49:40.752054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:40.752034    5419 scope.go:122] "RemoveContainer" containerID="c05e7e3bf658d0cf88180995ff05b55ce1fa6d88702b4b1d3153cc152a153eaf"
Oct 31 01:49:40.761843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:40.761823    5419 scope.go:122] "RemoveContainer" containerID="f71cef98421db23a50e3e62865d5eac77a540a2e6c7d888fd74696a3c8b40c80"
Oct 31 01:49:40.770947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:40.770922    5419 scope.go:122] "RemoveContainer" containerID="25e0441b981b6f086c941e5f68b9016c1f6b0e97dae085e35b6950a52996d430"
Oct 31 01:49:41.581576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.581537    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"]
Oct 31 01:49:41.582228 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582209    5419 cpu_manager.go:414] "containerMap: removing container" podUID="578c5687-d459-454e-b2c2-5567165e8945" containerName="payload-processing"
Oct 31 01:49:41.582287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582230    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578c5687-d459-454e-b2c2-5567165e8945" containerName="payload-processing"
Oct 31 01:49:41.582287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582275    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a43872e4-7f61-4d55-a513-64d2c542e378" containerName="revoke-keys"
Oct 31 01:49:41.582287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582280    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43872e4-7f61-4d55-a513-64d2c542e378" containerName="revoke-keys"
Oct 31 01:49:41.582388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582291    5419 cpu_manager.go:414] "containerMap: removing container" podUID="644432a3-30e0-42eb-bf49-f53cc9a4f1b5" containerName="payload-pre-processing"
Oct 31 01:49:41.582388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582296    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644432a3-30e0-42eb-bf49-f53cc9a4f1b5" containerName="payload-pre-processing"
Oct 31 01:49:41.582453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582397    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43872e4-7f61-4d55-a513-64d2c542e378" containerName="revoke-keys"
Oct 31 01:49:41.582453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582407    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="644432a3-30e0-42eb-bf49-f53cc9a4f1b5" containerName="payload-pre-processing"
Oct 31 01:49:41.582453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.582417    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="578c5687-d459-454e-b2c2-5567165e8945" containerName="payload-processing"
Oct 31 01:49:41.587161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.587129    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.642417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.642375    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"]
Oct 31 01:49:41.663491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.663450    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.663694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.663519    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqrnd\" (UniqueName: \"kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.663824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.663799    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.688051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.688010    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:49:41.688295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.688070    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:49:41.688295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.688182    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:49:41.688811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.688793    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:49:41.765523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.765476    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqrnd\" (UniqueName: \"kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.765960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.765721    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.765960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.765805    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.766580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.766518    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.768529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.768505    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.822347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.822310    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqrnd\" (UniqueName: \"kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd\") pod \"maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:41.902249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:41.902163    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:42.286459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:42.286428    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"]
Oct 31 01:49:42.288310 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:49:42.288280    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050a1256_2dd9_4bca_947c_d57e6654eec6.slice/crio-d773c6267cdc8a749ba8b1b6598b039816f5d8a25ad7f2abbaf6556eb57fe357 WatchSource:0}: Error finding container d773c6267cdc8a749ba8b1b6598b039816f5d8a25ad7f2abbaf6556eb57fe357: Status 404 returned error can't find the container with id d773c6267cdc8a749ba8b1b6598b039816f5d8a25ad7f2abbaf6556eb57fe357
Oct 31 01:49:43.194542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:43.194502    5419 generic.go:356] "Generic (PLEG): container finished" podID="050a1256-2dd9-4bca-947c-d57e6654eec6" containerID="5079b29c3c09676020a34794579ba453cde1231c2729c5b948d146315459a22c" exitCode=0
Oct 31 01:49:43.194941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:43.194538    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm" event={"ID":"050a1256-2dd9-4bca-947c-d57e6654eec6","Type":"ContainerDied","Data":"5079b29c3c09676020a34794579ba453cde1231c2729c5b948d146315459a22c"}
Oct 31 01:49:43.194941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:43.194592    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm" event={"ID":"050a1256-2dd9-4bca-947c-d57e6654eec6","Type":"ContainerStarted","Data":"d773c6267cdc8a749ba8b1b6598b039816f5d8a25ad7f2abbaf6556eb57fe357"}
Oct 31 01:49:44.202665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.202578    5419 generic.go:356] "Generic (PLEG): container finished" podID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerID="4ff5f0d5a424e6a567972a20e9ce95d212ca78c2cb589557587025fa9644933e" exitCode=0
Oct 31 01:49:44.202665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.202654    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" event={"ID":"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97","Type":"ContainerDied","Data":"4ff5f0d5a424e6a567972a20e9ce95d212ca78c2cb589557587025fa9644933e"}
Oct 31 01:49:44.203249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.203064    5419 scope.go:122] "RemoveContainer" containerID="4ff5f0d5a424e6a567972a20e9ce95d212ca78c2cb589557587025fa9644933e"
Oct 31 01:49:44.204681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.204652    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:44.204796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.204717    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:44.204796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.204748    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:49:44.370448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.370417    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:44.397943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.397901    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca\") pod \"050a1256-2dd9-4bca-947c-d57e6654eec6\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") "
Oct 31 01:49:44.398130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.398111    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca\") pod \"050a1256-2dd9-4bca-947c-d57e6654eec6\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") "
Oct 31 01:49:44.398300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.398277    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd\" (UniqueName: \"kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd\") pod \"050a1256-2dd9-4bca-947c-d57e6654eec6\" (UID: \"050a1256-2dd9-4bca-947c-d57e6654eec6\") "
Oct 31 01:49:44.399545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.399513    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca" pod "050a1256-2dd9-4bca-947c-d57e6654eec6" (UID: "050a1256-2dd9-4bca-947c-d57e6654eec6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:44.401711 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.401682    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd" pod "050a1256-2dd9-4bca-947c-d57e6654eec6" (UID: "050a1256-2dd9-4bca-947c-d57e6654eec6"). InnerVolumeSpecName "kube-api-access-mqrnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:44.401831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.401804    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca" pod "050a1256-2dd9-4bca-947c-d57e6654eec6" (UID: "050a1256-2dd9-4bca-947c-d57e6654eec6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:49:44.499899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.499867    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqrnd\" (UniqueName: \"kubernetes.io/projected/050a1256-2dd9-4bca-947c-d57e6654eec6-kube-api-access-mqrnd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:44.499899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.499895    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:44.500099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:44.499907    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050a1256-2dd9-4bca-947c-d57e6654eec6-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:44.582838 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:49:44.582757    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:49:45.240191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.240152    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm" event={"ID":"050a1256-2dd9-4bca-947c-d57e6654eec6","Type":"ContainerDied","Data":"d773c6267cdc8a749ba8b1b6598b039816f5d8a25ad7f2abbaf6556eb57fe357"}
Oct 31 01:49:45.240802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.240781    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d773c6267cdc8a749ba8b1b6598b039816f5d8a25ad7f2abbaf6556eb57fe357"
Oct 31 01:49:45.241033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.240972    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"
Oct 31 01:49:45.248183 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.248117    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" event={"ID":"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97","Type":"ContainerStarted","Data":"ec4a24e698089ebb6ebd34740397fcb9634b3dc96f8b94ca4d6c1a93c4e8ba33"}
Oct 31 01:49:45.355490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.355457    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:49:45.355691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.355662    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:49:45.355741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:45.355705    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:49:46.355645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:46.355612    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:49:46.356031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:46.355661    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:49:47.355173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:47.355102    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:49:47.355374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:47.355201    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:49:48.355495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:48.355463    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:49:48.355918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:48.355515    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:49:49.355948 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:49.355915    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:49:49.356389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:49.355971    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:49:50.356025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:50.355982    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 31 01:49:50.356467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:50.356055    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 31 01:49:51.360336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:51.360306    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:49:51.360703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:51.360577    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:49:51.361332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:51.361316    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:49:53.308574 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:53.308540    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"]
Oct 31 01:49:53.412040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:53.412010    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"]
Oct 31 01:49:58.309610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.309539    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw" podUID="76a1d415-5aa1-4e21-b60c-e72a97b2dab0" containerName="payload-processing" containerID="cri-o://22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7" gracePeriod=25
Oct 31 01:49:58.412376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.412336    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6" podUID="76e7406b-eec1-4856-88ed-50634b5b9bc8" containerName="payload-pre-processing" containerID="cri-o://e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900" gracePeriod=25
Oct 31 01:49:58.615997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.615911    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:49:58.654260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.654230    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn\" (UniqueName: \"kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn\") pod \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") "
Oct 31 01:49:58.654421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.654295    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume\") pod \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\" (UID: \"76a1d415-5aa1-4e21-b60c-e72a97b2dab0\") "
Oct 31 01:49:58.654718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.654691    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume" pod "76a1d415-5aa1-4e21-b60c-e72a97b2dab0" (UID: "76a1d415-5aa1-4e21-b60c-e72a97b2dab0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:58.654937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.654886    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:58.656389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.656365    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn" pod "76a1d415-5aa1-4e21-b60c-e72a97b2dab0" (UID: "76a1d415-5aa1-4e21-b60c-e72a97b2dab0"). InnerVolumeSpecName "kube-api-access-7lrsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:58.671399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.671376    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:49:58.755282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.755250    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume\") pod \"76e7406b-eec1-4856-88ed-50634b5b9bc8\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") "
Oct 31 01:49:58.755492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.755401    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8\" (UniqueName: \"kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8\") pod \"76e7406b-eec1-4856-88ed-50634b5b9bc8\" (UID: \"76e7406b-eec1-4856-88ed-50634b5b9bc8\") "
Oct 31 01:49:58.755697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.755664    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume" pod "76e7406b-eec1-4856-88ed-50634b5b9bc8" (UID: "76e7406b-eec1-4856-88ed-50634b5b9bc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:49:58.755827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.755760    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrsn\" (UniqueName: \"kubernetes.io/projected/76a1d415-5aa1-4e21-b60c-e72a97b2dab0-kube-api-access-7lrsn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:58.757595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.757574    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8" pod "76e7406b-eec1-4856-88ed-50634b5b9bc8" (UID: "76e7406b-eec1-4856-88ed-50634b5b9bc8"). InnerVolumeSpecName "kube-api-access-kwrf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:49:58.860932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.860905    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwrf8\" (UniqueName: \"kubernetes.io/projected/76e7406b-eec1-4856-88ed-50634b5b9bc8-kube-api-access-kwrf8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:58.860932 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:58.860929    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76e7406b-eec1-4856-88ed-50634b5b9bc8-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:49:59.346691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.346660    5419 generic.go:356] "Generic (PLEG): container finished" podID="76a1d415-5aa1-4e21-b60c-e72a97b2dab0" containerID="22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7" exitCode=0
Oct 31 01:49:59.347089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.346722    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"
Oct 31 01:49:59.347089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.346745    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw" event={"ID":"76a1d415-5aa1-4e21-b60c-e72a97b2dab0","Type":"ContainerDied","Data":"22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7"}
Oct 31 01:49:59.347089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.346838    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw" event={"ID":"76a1d415-5aa1-4e21-b60c-e72a97b2dab0","Type":"ContainerDied","Data":"f18c2e1cf392cad21c12a467234d9c3296d31f64837e777bab2a6d79ea102ec0"}
Oct 31 01:49:59.347089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.346867    5419 scope.go:122] "RemoveContainer" containerID="22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7"
Oct 31 01:49:59.348197 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.348160    5419 generic.go:356] "Generic (PLEG): container finished" podID="76e7406b-eec1-4856-88ed-50634b5b9bc8" containerID="e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900" exitCode=0
Oct 31 01:49:59.348288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.348236    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"
Oct 31 01:49:59.348288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.348279    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6" event={"ID":"76e7406b-eec1-4856-88ed-50634b5b9bc8","Type":"ContainerDied","Data":"e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900"}
Oct 31 01:49:59.348376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.348320    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6" event={"ID":"76e7406b-eec1-4856-88ed-50634b5b9bc8","Type":"ContainerDied","Data":"1cc7eeb58d8b2632ce5d94f2f7ebf2664c0b1d4676767813ae35c798c4b144aa"}
Oct 31 01:49:59.357066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.357045    5419 scope.go:122] "RemoveContainer" containerID="22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7"
Oct 31 01:49:59.357355 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:49:59.357335    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7\": container with ID starting with 22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7 not found: ID does not exist" containerID="22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7"
Oct 31 01:49:59.357432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.357366    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7"} err="failed to get container status \"22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7\": rpc error: code = NotFound desc = could not find container \"22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7\": container with ID starting with 22477ad824ca282cf0bbc5206593bc4b444c03201dcca173d9d9d8f11906b1e7 not found: ID does not exist"
Oct 31 01:49:59.357432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.357382    5419 scope.go:122] "RemoveContainer" containerID="e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900"
Oct 31 01:49:59.366391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.366365    5419 scope.go:122] "RemoveContainer" containerID="e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900"
Oct 31 01:49:59.366636 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:49:59.366619    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900\": container with ID starting with e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900 not found: ID does not exist" containerID="e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900"
Oct 31 01:49:59.366703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.366643    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900"} err="failed to get container status \"e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900\": rpc error: code = NotFound desc = could not find container \"e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900\": container with ID starting with e7b5fe646b4e361b6c0942fdfa0480aa5b1e767cf01408f2157b3f3092f14900 not found: ID does not exist"
Oct 31 01:49:59.511602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.511565    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"]
Oct 31 01:49:59.560958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.560925    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c116f3-5d65cdbbcf-rhnk6"]
Oct 31 01:49:59.621496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.621400    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"]
Oct 31 01:49:59.624618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:49:59.624594    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c116f3-78c68787f-dvczw"]
Oct 31 01:50:00.838115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:00.838082    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a1d415-5aa1-4e21-b60c-e72a97b2dab0" path="/var/lib/kubelet/pods/76a1d415-5aa1-4e21-b60c-e72a97b2dab0/volumes"
Oct 31 01:50:00.838536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:00.838522    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e7406b-eec1-4856-88ed-50634b5b9bc8" path="/var/lib/kubelet/pods/76e7406b-eec1-4856-88ed-50634b5b9bc8/volumes"
Oct 31 01:50:03.753315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:03.753272    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:50:14.582402 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:14.582362    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:50:23.747229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747195    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"]
Oct 31 01:50:23.747794 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747778    5419 cpu_manager.go:414] "containerMap: removing container" podUID="76e7406b-eec1-4856-88ed-50634b5b9bc8" containerName="payload-pre-processing"
Oct 31 01:50:23.747842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747795    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e7406b-eec1-4856-88ed-50634b5b9bc8" containerName="payload-pre-processing"
Oct 31 01:50:23.747842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747820    5419 cpu_manager.go:414] "containerMap: removing container" podUID="76a1d415-5aa1-4e21-b60c-e72a97b2dab0" containerName="payload-processing"
Oct 31 01:50:23.747842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747825    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a1d415-5aa1-4e21-b60c-e72a97b2dab0" containerName="payload-processing"
Oct 31 01:50:23.747842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747843    5419 cpu_manager.go:414] "containerMap: removing container" podUID="050a1256-2dd9-4bca-947c-d57e6654eec6" containerName="revoke-keys"
Oct 31 01:50:23.747971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747848    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050a1256-2dd9-4bca-947c-d57e6654eec6" containerName="revoke-keys"
Oct 31 01:50:23.747971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747952    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a1d415-5aa1-4e21-b60c-e72a97b2dab0" containerName="payload-processing"
Oct 31 01:50:23.747971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747962    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="050a1256-2dd9-4bca-947c-d57e6654eec6" containerName="revoke-keys"
Oct 31 01:50:23.748086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.747973    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e7406b-eec1-4856-88ed-50634b5b9bc8" containerName="payload-pre-processing"
Oct 31 01:50:23.751528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.751512    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.794922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.794889    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw78r\" (UniqueName: \"kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.795100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.794958    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.795178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.795155    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.795272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.795247    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.795343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.795290    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.795343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.795337    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.854096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.854063    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c0daf4\"/\"kube-root-ca.crt\""
Oct 31 01:50:23.854338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.854125    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c0daf4\"/\"default-dockercfg-xhp4g\""
Oct 31 01:50:23.854338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.854168    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c0daf4\"/\"openshift-service-ca.crt\""
Oct 31 01:50:23.854338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.854183    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c0daf4\"/\"praxis-route-c0daf4-kserve-self-signed-certs\""
Oct 31 01:50:23.856901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.856879    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"]
Oct 31 01:50:23.910161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.907990    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.910161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.908059    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.910161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.908090    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.910161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.908132    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.910161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.908252    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78r\" (UniqueName: \"kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.910161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.908316    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.911612 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.911565    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.919161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.911880    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.919161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.912569    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.943519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.943485    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.951236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.946780    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:23.969420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:23.969393    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw78r\" (UniqueName: \"kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r\") pod \"praxis-route-c0daf4-kserve-69c46b7545-2mqw2\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") " pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:24.064067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:24.063982    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:24.439311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:24.439281    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"]
Oct 31 01:50:24.440983 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:50:24.440956    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbce7ceea_4c5a_4cf7_8d69_887ce9c92ed3.slice/crio-f9eb1ce820e2213d98f6f4ebb5db1e0e45108f83c7c67ab04aee42d091548892 WatchSource:0}: Error finding container f9eb1ce820e2213d98f6f4ebb5db1e0e45108f83c7c67ab04aee42d091548892: Status 404 returned error can't find the container with id f9eb1ce820e2213d98f6f4ebb5db1e0e45108f83c7c67ab04aee42d091548892
Oct 31 01:50:24.512110 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:24.512083    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" event={"ID":"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3","Type":"ContainerStarted","Data":"17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940"}
Oct 31 01:50:24.512300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:24.512123    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" event={"ID":"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3","Type":"ContainerStarted","Data":"f9eb1ce820e2213d98f6f4ebb5db1e0e45108f83c7c67ab04aee42d091548892"}
Oct 31 01:50:24.627316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:24.627262    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" podStartSLOduration=1.627249406 podStartE2EDuration="1.627249406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:24.625398746 +0000 UTC m=+2745.926355337" watchObservedRunningTime="2026-10-31 01:50:24.627249406 +0000 UTC m=+2745.928205983"
Oct 31 01:50:25.064546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:25.064518    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:27.502337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:27.502304    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"]
Oct 31 01:50:27.551666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:27.551634    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-8a3796-wcflj"]
Oct 31 01:50:28.084219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.084185    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"]
Oct 31 01:50:28.084425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.084394    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx" podUID="765496b6-57f5-419b-bec5-93a7758586f3" containerName="payload-processing" containerID="cri-o://7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33" gracePeriod=30
Oct 31 01:50:28.138094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.138067    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"]
Oct 31 01:50:28.138334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.138310    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt" podUID="68670ba5-7785-4c9e-bdd0-f61a916c5c01" containerName="payload-pre-processing" containerID="cri-o://116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b" gracePeriod=30
Oct 31 01:50:28.442909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.442885    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:50:28.445551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.445530    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:50:28.538578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.538547    5419 generic.go:356] "Generic (PLEG): container finished" podID="68670ba5-7785-4c9e-bdd0-f61a916c5c01" containerID="116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b" exitCode=0
Oct 31 01:50:28.539002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.538586    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt" event={"ID":"68670ba5-7785-4c9e-bdd0-f61a916c5c01","Type":"ContainerDied","Data":"116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b"}
Oct 31 01:50:28.539002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.538639    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt" event={"ID":"68670ba5-7785-4c9e-bdd0-f61a916c5c01","Type":"ContainerDied","Data":"b391892eaa9822ed322a2458ffe31f5cb6bb958f40a82b3537e65c71fc25d771"}
Oct 31 01:50:28.539002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.538663    5419 scope.go:122] "RemoveContainer" containerID="116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b"
Oct 31 01:50:28.539002 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.538593    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"
Oct 31 01:50:28.540104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.540085    5419 generic.go:356] "Generic (PLEG): container finished" podID="765496b6-57f5-419b-bec5-93a7758586f3" containerID="7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33" exitCode=0
Oct 31 01:50:28.540226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.540129    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"
Oct 31 01:50:28.540226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.540190    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx" event={"ID":"765496b6-57f5-419b-bec5-93a7758586f3","Type":"ContainerDied","Data":"7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33"}
Oct 31 01:50:28.540307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.540247    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx" event={"ID":"765496b6-57f5-419b-bec5-93a7758586f3","Type":"ContainerDied","Data":"036d4e966c49688465aab02a509f6704dc792bc6074c0f442db8008df1739c90"}
Oct 31 01:50:28.546798 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.546777    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume\") pod \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") "
Oct 31 01:50:28.546943 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.546912    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9\" (UniqueName: \"kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9\") pod \"765496b6-57f5-419b-bec5-93a7758586f3\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") "
Oct 31 01:50:28.547007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.546954    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d\" (UniqueName: \"kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d\") pod \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\" (UID: \"68670ba5-7785-4c9e-bdd0-f61a916c5c01\") "
Oct 31 01:50:28.547007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.546997    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume\") pod \"765496b6-57f5-419b-bec5-93a7758586f3\" (UID: \"765496b6-57f5-419b-bec5-93a7758586f3\") "
Oct 31 01:50:28.547357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.547225    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume" pod "68670ba5-7785-4c9e-bdd0-f61a916c5c01" (UID: "68670ba5-7785-4c9e-bdd0-f61a916c5c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:28.547482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.547396    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume" pod "765496b6-57f5-419b-bec5-93a7758586f3" (UID: "765496b6-57f5-419b-bec5-93a7758586f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:28.549167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.549122    5419 scope.go:122] "RemoveContainer" containerID="116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b"
Oct 31 01:50:28.549291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.549272    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9" pod "765496b6-57f5-419b-bec5-93a7758586f3" (UID: "765496b6-57f5-419b-bec5-93a7758586f3"). InnerVolumeSpecName "kube-api-access-zk5c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:28.549513 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:28.549492    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b\": container with ID starting with 116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b not found: ID does not exist" containerID="116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b"
Oct 31 01:50:28.549513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.549504    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d" pod "68670ba5-7785-4c9e-bdd0-f61a916c5c01" (UID: "68670ba5-7785-4c9e-bdd0-f61a916c5c01"). InnerVolumeSpecName "kube-api-access-bfp6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:28.549644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.549524    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b"} err="failed to get container status \"116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b\": rpc error: code = NotFound desc = could not find container \"116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b\": container with ID starting with 116d555b95e62894852b700bfbe034727dce26b93a47deb100538d8a9a02229b not found: ID does not exist"
Oct 31 01:50:28.549644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.549550    5419 scope.go:122] "RemoveContainer" containerID="7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33"
Oct 31 01:50:28.646227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.644317    5419 scope.go:122] "RemoveContainer" containerID="7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33"
Oct 31 01:50:28.654065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.653665    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5c9\" (UniqueName: \"kubernetes.io/projected/765496b6-57f5-419b-bec5-93a7758586f3-kube-api-access-zk5c9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.654065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.653703    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfp6d\" (UniqueName: \"kubernetes.io/projected/68670ba5-7785-4c9e-bdd0-f61a916c5c01-kube-api-access-bfp6d\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.654065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.653719    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765496b6-57f5-419b-bec5-93a7758586f3-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.654065 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.653733    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68670ba5-7785-4c9e-bdd0-f61a916c5c01-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:28.660460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.660394    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"]
Oct 31 01:50:28.661672 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:28.661572    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33\": container with ID starting with 7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33 not found: ID does not exist" containerID="7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33"
Oct 31 01:50:28.661672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.661619    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33"} err="failed to get container status \"7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33\": rpc error: code = NotFound desc = could not find container \"7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33\": container with ID starting with 7c97b7715b5fa6ea71007fe9e0cb11eef85d01e6304394757622220722055f33 not found: ID does not exist"
Oct 31 01:50:28.666374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.666341    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c116f3-5vfsm"]
Oct 31 01:50:28.842511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.842478    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050a1256-2dd9-4bca-947c-d57e6654eec6" path="/var/lib/kubelet/pods/050a1256-2dd9-4bca-947c-d57e6654eec6/volumes"
Oct 31 01:50:28.843185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.843165    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43872e4-7f61-4d55-a513-64d2c542e378" path="/var/lib/kubelet/pods/a43872e4-7f61-4d55-a513-64d2c542e378/volumes"
Oct 31 01:50:28.962202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.962092    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"]
Oct 31 01:50:28.965720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:28.965690    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-53af08-5f4977594c-xwmrt"]
Oct 31 01:50:29.073738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:29.073705    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"]
Oct 31 01:50:29.078127 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:29.078089    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-53af08-65d5bc6896-8bjdx"]
Oct 31 01:50:29.964423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:29.964393    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"]
Oct 31 01:50:29.967364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:29.967339    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-53af08-cpc74"]
Oct 31 01:50:30.837279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:30.837245    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68670ba5-7785-4c9e-bdd0-f61a916c5c01" path="/var/lib/kubelet/pods/68670ba5-7785-4c9e-bdd0-f61a916c5c01/volumes"
Oct 31 01:50:30.837713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:30.837698    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765496b6-57f5-419b-bec5-93a7758586f3" path="/var/lib/kubelet/pods/765496b6-57f5-419b-bec5-93a7758586f3/volumes"
Oct 31 01:50:30.838093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:30.838080    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d173dd6-cd28-4cd3-a319-9900221763af" path="/var/lib/kubelet/pods/9d173dd6-cd28-4cd3-a319-9900221763af/volumes"
Oct 31 01:50:34.064927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:34.064892    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:34.077318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:34.077290    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:34.592193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:34.592159    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:50:35.098836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:35.098791    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"]
Oct 31 01:50:35.099682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:35.099624    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy" containerID="cri-o://da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed" gracePeriod=30
Oct 31 01:50:35.102676 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:35.102640    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"]
Oct 31 01:50:35.102869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:35.102844    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" containerID="cri-o://cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90" gracePeriod=30
Oct 31 01:50:39.875897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:39.875854    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:50:39.876343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:39.875926    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:50:40.399254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.399232    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:50:40.414784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.414759    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:50:40.533940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.533912    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.533940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.533957    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.534193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.533990    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.534193 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534077    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.534395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534200    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534241    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534269    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534336    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.534620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534457    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.534620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534519    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534530    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534577    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.534851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534652    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534704    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534736    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.534851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534768    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.534851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534799    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.535117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534871    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.535117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534907    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs\") pod \"0a8c54e6-3039-41c0-af19-75ca5be63d52\" (UID: \"0a8c54e6-3039-41c0-af19-75ca5be63d52\") "
Oct 31 01:50:40.535117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.534950    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert\") pod \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\" (UID: \"0c554ef4-56d2-4e7d-bf35-69c1e5706d56\") "
Oct 31 01:50:40.535293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535119    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.535349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535318    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.535790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535590    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:40.535790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535765    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.535790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535779    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535803    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535816    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535838    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535854    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535861    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535868    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a8c54e6-3039-41c0-af19-75ca5be63d52-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.536048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.535885    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.537095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537061    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.537095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537069    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.537293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537095    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:40.537293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537115    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537510    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp" pod "0c554ef4-56d2-4e7d-bf35-69c1e5706d56" (UID: "0c554ef4-56d2-4e7d-bf35-69c1e5706d56"). InnerVolumeSpecName "kube-api-access-28pnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537562    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "kube-api-access-8cgz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.537825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.537806    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:40.538284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.538266    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:50:40.538741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.538716    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo" pod "0a8c54e6-3039-41c0-af19-75ca5be63d52" (UID: "0a8c54e6-3039-41c0-af19-75ca5be63d52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:50:40.623569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.623533    5419 generic.go:356] "Generic (PLEG): container finished" podID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerID="da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed" exitCode=0
Oct 31 01:50:40.623811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.623584    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"
Oct 31 01:50:40.623811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.623619    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" event={"ID":"0c554ef4-56d2-4e7d-bf35-69c1e5706d56","Type":"ContainerDied","Data":"da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed"}
Oct 31 01:50:40.623811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.623674    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj" event={"ID":"0c554ef4-56d2-4e7d-bf35-69c1e5706d56","Type":"ContainerDied","Data":"842c0b39dc94800a4611fb7e195442ebb2666fd81b55e4c75163583bdc25dc3c"}
Oct 31 01:50:40.623811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.623705    5419 scope.go:122] "RemoveContainer" containerID="da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed"
Oct 31 01:50:40.626073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.626043    5419 generic.go:356] "Generic (PLEG): container finished" podID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerID="cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90" exitCode=0
Oct 31 01:50:40.626244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.626102    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"
Oct 31 01:50:40.626244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.626108    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" event={"ID":"0a8c54e6-3039-41c0-af19-75ca5be63d52","Type":"ContainerDied","Data":"cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90"}
Oct 31 01:50:40.626244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.626229    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj" event={"ID":"0a8c54e6-3039-41c0-af19-75ca5be63d52","Type":"ContainerDied","Data":"a8d24640df4a96a00e3900fbb1a5f097563d148eaecceeb9985e01f3970cd460"}
Oct 31 01:50:40.636035 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636014    5419 scope.go:122] "RemoveContainer" containerID="e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"
Oct 31 01:50:40.636813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636786    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636821    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28pnp\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-kube-api-access-28pnp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636840    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636855    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636871    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgz2\" (UniqueName: \"kubernetes.io/projected/0a8c54e6-3039-41c0-af19-75ca5be63d52-kube-api-access-8cgz2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636886    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636900    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.636914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636915    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636930    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636945    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8c54e6-3039-41c0-af19-75ca5be63d52-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636957    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636969    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.637187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.636982    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c554ef4-56d2-4e7d-bf35-69c1e5706d56-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:40.645626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.645608    5419 scope.go:122] "RemoveContainer" containerID="da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed"
Oct 31 01:50:40.645923 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:40.645905    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed\": container with ID starting with da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed not found: ID does not exist" containerID="da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed"
Oct 31 01:50:40.645979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.645931    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed"} err="failed to get container status \"da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed\": rpc error: code = NotFound desc = could not find container \"da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed\": container with ID starting with da68be93e55796f91d96c00f67dee34551f51390f3a65da5f7cbecd90b4abaed not found: ID does not exist"
Oct 31 01:50:40.645979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.645953    5419 scope.go:122] "RemoveContainer" containerID="e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"
Oct 31 01:50:40.646234 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:40.646215    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e\": container with ID starting with e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e not found: ID does not exist" containerID="e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"
Oct 31 01:50:40.646318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.646244    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e"} err="failed to get container status \"e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e\": rpc error: code = NotFound desc = could not find container \"e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e\": container with ID starting with e613612b17224f40952e46ee95bc84292aaf598b7d2303e364da768d3d7d0a2e not found: ID does not exist"
Oct 31 01:50:40.646318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.646269    5419 scope.go:122] "RemoveContainer" containerID="cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90"
Oct 31 01:50:40.655784 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.655761    5419 scope.go:122] "RemoveContainer" containerID="cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90"
Oct 31 01:50:40.656077 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:40.656055    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90\": container with ID starting with cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90 not found: ID does not exist" containerID="cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90"
Oct 31 01:50:40.656182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.656087    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90"} err="failed to get container status \"cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90\": rpc error: code = NotFound desc = could not find container \"cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90\": container with ID starting with cd2351892769ad0e87a876b7a76c12bb21feb43ff30008243edf6247f03dfb90 not found: ID does not exist"
Oct 31 01:50:40.801628 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.801586    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"]
Oct 31 01:50:40.801846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.801650    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-s2spj"]
Oct 31 01:50:40.818393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.818367    5419 scope.go:122] "RemoveContainer" containerID="cbe12d1e1c5462d575b6989e364ee49f0e410d643e55fba31c4779b687af6487"
Oct 31 01:50:40.828000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.827974    5419 scope.go:122] "RemoveContainer" containerID="6f93c1fbbe00ce230ea42423dffe697e159442c0fe1e33ace2bb59bd6eb9677b"
Oct 31 01:50:40.841081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.841056    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" path="/var/lib/kubelet/pods/0c554ef4-56d2-4e7d-bf35-69c1e5706d56/volumes"
Oct 31 01:50:40.905713 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.905641    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"]
Oct 31 01:50:40.909484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:40.909460    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c116f3-openshift-default-cf4d85cb9-2k8qj"]
Oct 31 01:50:42.837566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:42.837534    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" path="/var/lib/kubelet/pods/0a8c54e6-3039-41c0-af19-75ca5be63d52/volumes"
Oct 31 01:50:44.603948 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:50:44.603911    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:50:44.838575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.838544    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:50:44.881838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.881760    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"]
Oct 31 01:50:44.882406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882390    5419 cpu_manager.go:414] "containerMap: removing container" podUID="68670ba5-7785-4c9e-bdd0-f61a916c5c01" containerName="payload-pre-processing"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882407    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68670ba5-7785-4c9e-bdd0-f61a916c5c01" containerName="payload-pre-processing"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882435    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882441    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882464    5419 cpu_manager.go:414] "containerMap: removing container" podUID="765496b6-57f5-419b-bec5-93a7758586f3" containerName="payload-processing"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882469    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765496b6-57f5-419b-bec5-93a7758586f3" containerName="payload-processing"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882484    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882489    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882499    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy"
Oct 31 01:50:44.882492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882503    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy"
Oct 31 01:50:44.882951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882588    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8c54e6-3039-41c0-af19-75ca5be63d52" containerName="istio-proxy"
Oct 31 01:50:44.882951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882600    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="765496b6-57f5-419b-bec5-93a7758586f3" containerName="payload-processing"
Oct 31 01:50:44.882951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882609    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy"
Oct 31 01:50:44.882951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882621    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c554ef4-56d2-4e7d-bf35-69c1e5706d56" containerName="istio-proxy"
Oct 31 01:50:44.882951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.882632    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="68670ba5-7785-4c9e-bdd0-f61a916c5c01" containerName="payload-pre-processing"
Oct 31 01:50:44.906394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.904283    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:44.953771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.953734    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"]
Oct 31 01:50:44.980941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.980902    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:50:44.981115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.981027    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:50:44.989514 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.989493    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:50:44.989633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:44.989527    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:50:45.017804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.017773    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx2zl\" (UniqueName: \"kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.017977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.017833    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.017977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.017901    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.119291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.119251    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx2zl\" (UniqueName: \"kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.119494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.119320    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.119494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.119353    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.120192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.120164    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.121976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.121954    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.175908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.175833    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx2zl\" (UniqueName: \"kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl\") pod \"maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.250559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.250504    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:45.698559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.698535    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"]
Oct 31 01:50:45.701109 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:50:45.701076    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2c55e8_afe5_4e6b_8f70_afacc0a1c440.slice/crio-d2dc04748765adf6d178b516976f3f640a01d73fec608f9aacd76d9eefc7564e WatchSource:0}: Error finding container d2dc04748765adf6d178b516976f3f640a01d73fec608f9aacd76d9eefc7564e: Status 404 returned error can't find the container with id d2dc04748765adf6d178b516976f3f640a01d73fec608f9aacd76d9eefc7564e
Oct 31 01:50:45.981015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.980988    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"]
Oct 31 01:50:45.986523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:45.986506    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.106282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.106237    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"]
Oct 31 01:50:46.151986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.151945    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2rw\" (UniqueName: \"kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.152200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.152010    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.152200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.152118    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.253778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.253684    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.253778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.253736    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2rw\" (UniqueName: \"kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.253778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.253778    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.254420 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.254394    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.256281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.256258    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.310887 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.310856    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2rw\" (UniqueName: \"kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw\") pod \"maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.600892 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.600810    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:46.673823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.673715    5419 generic.go:356] "Generic (PLEG): container finished" podID="0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" containerID="327ff0cbf32457aefc8556ae830eccf087abf8ca5eb24e8dd7f4045b2cd1d97f" exitCode=0
Oct 31 01:50:46.673823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.673755    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt" event={"ID":"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440","Type":"ContainerDied","Data":"327ff0cbf32457aefc8556ae830eccf087abf8ca5eb24e8dd7f4045b2cd1d97f"}
Oct 31 01:50:46.673823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.673797    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt" event={"ID":"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440","Type":"ContainerStarted","Data":"d2dc04748765adf6d178b516976f3f640a01d73fec608f9aacd76d9eefc7564e"}
Oct 31 01:50:46.983295 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:46.983272    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"]
Oct 31 01:50:46.985011 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:50:46.984985    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecaf011_0207_42ef_854f_ef6caf47b699.slice/crio-dedb20dde038ce19bedf85e528e602f3be7c5f4340b5b8d6396d35e49638682f WatchSource:0}: Error finding container dedb20dde038ce19bedf85e528e602f3be7c5f4340b5b8d6396d35e49638682f: Status 404 returned error can't find the container with id dedb20dde038ce19bedf85e528e602f3be7c5f4340b5b8d6396d35e49638682f
Oct 31 01:50:47.682364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.682274    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j" event={"ID":"4ecaf011-0207-42ef-854f-ef6caf47b699","Type":"ContainerStarted","Data":"3104a2dae5d64b76aa5b233964fa9e8c1dad7434e099c3cef9dcc0da48ed73c5"}
Oct 31 01:50:47.682364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.682341    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j" event={"ID":"4ecaf011-0207-42ef-854f-ef6caf47b699","Type":"ContainerStarted","Data":"dedb20dde038ce19bedf85e528e602f3be7c5f4340b5b8d6396d35e49638682f"}
Oct 31 01:50:47.797810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.797748    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j" podStartSLOduration=2.797731171 podStartE2EDuration="2.797731171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:47.794343771 +0000 UTC m=+2769.095300348" watchObservedRunningTime="2026-10-31 01:50:47.797731171 +0000 UTC m=+2769.098687749"
Oct 31 01:50:47.830327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.830305    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:47.975086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.975052    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca\") pod \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") "
Oct 31 01:50:47.975316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.975131    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl\" (UniqueName: \"kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl\") pod \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") "
Oct 31 01:50:47.975316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.975212    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca\") pod \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\" (UID: \"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440\") "
Oct 31 01:50:47.975505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.975462    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca" pod "0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" (UID: "0c2c55e8-afe5-4e6b-8f70-afacc0a1c440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:47.976014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.975823    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:47.977439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.977412    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca" pod "0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" (UID: "0c2c55e8-afe5-4e6b-8f70-afacc0a1c440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:47.977537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:47.977515    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl" pod "0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" (UID: "0c2c55e8-afe5-4e6b-8f70-afacc0a1c440"). InnerVolumeSpecName "kube-api-access-cx2zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:48.080091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.078059    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx2zl\" (UniqueName: \"kubernetes.io/projected/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-kube-api-access-cx2zl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.080091 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.078096    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:48.690626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.690594    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"
Oct 31 01:50:48.690823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.690638    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt" event={"ID":"0c2c55e8-afe5-4e6b-8f70-afacc0a1c440","Type":"ContainerDied","Data":"d2dc04748765adf6d178b516976f3f640a01d73fec608f9aacd76d9eefc7564e"}
Oct 31 01:50:48.690823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.690714    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2dc04748765adf6d178b516976f3f640a01d73fec608f9aacd76d9eefc7564e"
Oct 31 01:50:48.692353 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.692329    5419 generic.go:356] "Generic (PLEG): container finished" podID="4ecaf011-0207-42ef-854f-ef6caf47b699" containerID="3104a2dae5d64b76aa5b233964fa9e8c1dad7434e099c3cef9dcc0da48ed73c5" exitCode=0
Oct 31 01:50:48.692465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:48.692395    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j" event={"ID":"4ecaf011-0207-42ef-854f-ef6caf47b699","Type":"ContainerDied","Data":"3104a2dae5d64b76aa5b233964fa9e8c1dad7434e099c3cef9dcc0da48ed73c5"}
Oct 31 01:50:49.848580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:49.848553    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:49.998027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:49.997995    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca\") pod \"4ecaf011-0207-42ef-854f-ef6caf47b699\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") "
Oct 31 01:50:49.998253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:49.998097    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw\" (UniqueName: \"kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw\") pod \"4ecaf011-0207-42ef-854f-ef6caf47b699\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") "
Oct 31 01:50:49.998253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:49.998191    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca\") pod \"4ecaf011-0207-42ef-854f-ef6caf47b699\" (UID: \"4ecaf011-0207-42ef-854f-ef6caf47b699\") "
Oct 31 01:50:49.998452 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:49.998428    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca" pod "4ecaf011-0207-42ef-854f-ef6caf47b699" (UID: "4ecaf011-0207-42ef-854f-ef6caf47b699"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:49.999229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:49.999191    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.000445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.000416    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw" pod "4ecaf011-0207-42ef-854f-ef6caf47b699" (UID: "4ecaf011-0207-42ef-854f-ef6caf47b699"). InnerVolumeSpecName "kube-api-access-hj2rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:50:50.000589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.000449    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca" pod "4ecaf011-0207-42ef-854f-ef6caf47b699" (UID: "4ecaf011-0207-42ef-854f-ef6caf47b699"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:50:50.100584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.100550    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ecaf011-0207-42ef-854f-ef6caf47b699-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.100584 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.100580    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj2rw\" (UniqueName: \"kubernetes.io/projected/4ecaf011-0207-42ef-854f-ef6caf47b699-kube-api-access-hj2rw\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:50.705883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.705848    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"
Oct 31 01:50:50.706073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.705884    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j" event={"ID":"4ecaf011-0207-42ef-854f-ef6caf47b699","Type":"ContainerDied","Data":"dedb20dde038ce19bedf85e528e602f3be7c5f4340b5b8d6396d35e49638682f"}
Oct 31 01:50:50.706073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:50.705933    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dedb20dde038ce19bedf85e528e602f3be7c5f4340b5b8d6396d35e49638682f"
Oct 31 01:50:56.712983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.712948    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"]
Oct 31 01:50:56.713511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.713486    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4ecaf011-0207-42ef-854f-ef6caf47b699" containerName="revoke-keys"
Oct 31 01:50:56.713511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.713498    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecaf011-0207-42ef-854f-ef6caf47b699" containerName="revoke-keys"
Oct 31 01:50:56.713598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.713517    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" containerName="revoke-keys"
Oct 31 01:50:56.713598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.713524    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" containerName="revoke-keys"
Oct 31 01:50:56.713675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.713626    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecaf011-0207-42ef-854f-ef6caf47b699" containerName="revoke-keys"
Oct 31 01:50:56.713675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.713637    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" containerName="revoke-keys"
Oct 31 01:50:56.717442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.717423    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.765559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.765522    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.765731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.765601    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.765829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.765808    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcx72\" (UniqueName: \"kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.778577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.778547    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"]
Oct 31 01:50:56.820582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.820552    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:50:56.820779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.820641    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:50:56.820779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.820715    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:50:56.820779 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.820743    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:50:56.867317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.867285    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.867505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.867333    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.867505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.867494    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcx72\" (UniqueName: \"kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.867990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.867969    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.869814 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.869793    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:56.923982 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:56.923956    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcx72\" (UniqueName: \"kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72\") pod \"maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:57.030894 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.030866    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:57.405342 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.405313    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"]
Oct 31 01:50:57.406840 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:50:57.406816    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b22bd41_0191_4f9b_ab69_7b8f7917e991.slice/crio-ff7f4973604ab592b940bcf95d872a8531a967191552e5722ed9dcef4de30555 WatchSource:0}: Error finding container ff7f4973604ab592b940bcf95d872a8531a967191552e5722ed9dcef4de30555: Status 404 returned error can't find the container with id ff7f4973604ab592b940bcf95d872a8531a967191552e5722ed9dcef4de30555
Oct 31 01:50:57.561643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.561612    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"]
Oct 31 01:50:57.672666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.672569    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"]
Oct 31 01:50:57.749178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.749131    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn" event={"ID":"5b22bd41-0191-4f9b-ab69-7b8f7917e991","Type":"ContainerStarted","Data":"6f53fbbe5d1689a32f9a20e6091ea3f6dec209a471902198a16c2ca912d41fa0"}
Oct 31 01:50:57.749613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.749205    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn" event={"ID":"5b22bd41-0191-4f9b-ab69-7b8f7917e991","Type":"ContainerStarted","Data":"ff7f4973604ab592b940bcf95d872a8531a967191552e5722ed9dcef4de30555"}
Oct 31 01:50:57.902705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:57.902655    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn" podStartSLOduration=1.902639206 podStartE2EDuration="1.902639206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:57.899804748 +0000 UTC m=+2779.200761326" watchObservedRunningTime="2026-10-31 01:50:57.902639206 +0000 UTC m=+2779.203595821"
Oct 31 01:50:58.757001 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:58.756966    5419 generic.go:356] "Generic (PLEG): container finished" podID="5b22bd41-0191-4f9b-ab69-7b8f7917e991" containerID="6f53fbbe5d1689a32f9a20e6091ea3f6dec209a471902198a16c2ca912d41fa0" exitCode=0
Oct 31 01:50:58.757506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:58.757053    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn" event={"ID":"5b22bd41-0191-4f9b-ab69-7b8f7917e991","Type":"ContainerDied","Data":"6f53fbbe5d1689a32f9a20e6091ea3f6dec209a471902198a16c2ca912d41fa0"}
Oct 31 01:50:59.917810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:59.917782    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:50:59.918250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:59.918067    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca\") pod \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") "
Oct 31 01:50:59.918250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:59.918109    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca\") pod \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") "
Oct 31 01:50:59.918505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:59.918481    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca" pod "5b22bd41-0191-4f9b-ab69-7b8f7917e991" (UID: "5b22bd41-0191-4f9b-ab69-7b8f7917e991"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:50:59.918589 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:59.918572    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:50:59.920430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:50:59.920409    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca" pod "5b22bd41-0191-4f9b-ab69-7b8f7917e991" (UID: "5b22bd41-0191-4f9b-ab69-7b8f7917e991"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:00.019639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.019543    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72\" (UniqueName: \"kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72\") pod \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\" (UID: \"5b22bd41-0191-4f9b-ab69-7b8f7917e991\") "
Oct 31 01:51:00.019951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.019932    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b22bd41-0191-4f9b-ab69-7b8f7917e991-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:00.021785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.021765    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72" pod "5b22bd41-0191-4f9b-ab69-7b8f7917e991" (UID: "5b22bd41-0191-4f9b-ab69-7b8f7917e991"). InnerVolumeSpecName "kube-api-access-wcx72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:00.120933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.120900    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcx72\" (UniqueName: \"kubernetes.io/projected/5b22bd41-0191-4f9b-ab69-7b8f7917e991-kube-api-access-wcx72\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:00.770269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.770231    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn" event={"ID":"5b22bd41-0191-4f9b-ab69-7b8f7917e991","Type":"ContainerDied","Data":"ff7f4973604ab592b940bcf95d872a8531a967191552e5722ed9dcef4de30555"}
Oct 31 01:51:00.770269 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.770264    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"
Oct 31 01:51:00.770522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:00.770289    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff7f4973604ab592b940bcf95d872a8531a967191552e5722ed9dcef4de30555"
Oct 31 01:51:02.562473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.562417    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh" podUID="2d2c2635-0d1d-49f3-bc35-eefe76a809b6" containerName="payload-processing" containerID="cri-o://65939587aacac51a90ac4eb4fbaafc588b518df322cce406c2a7a9143cea05c5" gracePeriod=25
Oct 31 01:51:02.688389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.680252    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph" podUID="8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" containerName="payload-pre-processing" containerID="cri-o://15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604" gracePeriod=25
Oct 31 01:51:02.789954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.789911    5419 generic.go:356] "Generic (PLEG): container finished" podID="2d2c2635-0d1d-49f3-bc35-eefe76a809b6" containerID="65939587aacac51a90ac4eb4fbaafc588b518df322cce406c2a7a9143cea05c5" exitCode=0
Oct 31 01:51:02.790119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.789983    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh" event={"ID":"2d2c2635-0d1d-49f3-bc35-eefe76a809b6","Type":"ContainerDied","Data":"65939587aacac51a90ac4eb4fbaafc588b518df322cce406c2a7a9143cea05c5"}
Oct 31 01:51:02.920500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.920354    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:51:02.988262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.988087    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz\" (UniqueName: \"kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz\") pod \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") "
Oct 31 01:51:02.988435 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.988297    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume\") pod \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\" (UID: \"2d2c2635-0d1d-49f3-bc35-eefe76a809b6\") "
Oct 31 01:51:02.991520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.988918    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume" pod "2d2c2635-0d1d-49f3-bc35-eefe76a809b6" (UID: "2d2c2635-0d1d-49f3-bc35-eefe76a809b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:02.992901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:02.992682    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz" pod "2d2c2635-0d1d-49f3-bc35-eefe76a809b6" (UID: "2d2c2635-0d1d-49f3-bc35-eefe76a809b6"). InnerVolumeSpecName "kube-api-access-7x9bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:03.052591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.052555    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:51:03.090215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.090188    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg\" (UniqueName: \"kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg\") pod \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") "
Oct 31 01:51:03.090458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.090354    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume\") pod \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\" (UID: \"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f\") "
Oct 31 01:51:03.090732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.090709    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume" pod "8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" (UID: "8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:03.090827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.090811    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:03.090873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.090830    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:03.090873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.090840    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x9bz\" (UniqueName: \"kubernetes.io/projected/2d2c2635-0d1d-49f3-bc35-eefe76a809b6-kube-api-access-7x9bz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:03.092415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.092395    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg" pod "8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" (UID: "8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f"). InnerVolumeSpecName "kube-api-access-2dmhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:03.192460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.192380    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dmhg\" (UniqueName: \"kubernetes.io/projected/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f-kube-api-access-2dmhg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:03.797731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.797700    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"
Oct 31 01:51:03.798206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.797702    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh" event={"ID":"2d2c2635-0d1d-49f3-bc35-eefe76a809b6","Type":"ContainerDied","Data":"c9f5d45af343210d08d28fa1539be46677f130a5832d0d4a3374c6d854fbf62e"}
Oct 31 01:51:03.798206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.797842    5419 scope.go:122] "RemoveContainer" containerID="65939587aacac51a90ac4eb4fbaafc588b518df322cce406c2a7a9143cea05c5"
Oct 31 01:51:03.799303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.799277    5419 generic.go:356] "Generic (PLEG): container finished" podID="8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" containerID="15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604" exitCode=0
Oct 31 01:51:03.799387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.799326    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"
Oct 31 01:51:03.799387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.799371    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph" event={"ID":"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f","Type":"ContainerDied","Data":"15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604"}
Oct 31 01:51:03.799510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.799437    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph" event={"ID":"8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f","Type":"ContainerDied","Data":"238b99501dfde7afbf221b9deabc62ccd7bbdb6daae6725369f3f4ad91524938"}
Oct 31 01:51:03.809652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.809617    5419 scope.go:122] "RemoveContainer" containerID="15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604"
Oct 31 01:51:03.818691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.818675    5419 scope.go:122] "RemoveContainer" containerID="15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604"
Oct 31 01:51:03.818958 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:51:03.818938    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604\": container with ID starting with 15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604 not found: ID does not exist" containerID="15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604"
Oct 31 01:51:03.819024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.818970    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604"} err="failed to get container status \"15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604\": rpc error: code = NotFound desc = could not find container \"15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604\": container with ID starting with 15fc04ae068b82601e2a752216ce4e93051d8e3f6d98bdc3b7a6da01639ba604 not found: ID does not exist"
Oct 31 01:51:03.961307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:03.961227    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"]
Oct 31 01:51:04.010716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:04.010683    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-e59f12-7dc7465c5c-25kvh"]
Oct 31 01:51:04.075616 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:04.075582    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"]
Oct 31 01:51:04.080656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:04.080631    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-e59f12-6cc6cd7775-kdcph"]
Oct 31 01:51:04.837445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:04.837416    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2c2635-0d1d-49f3-bc35-eefe76a809b6" path="/var/lib/kubelet/pods/2d2c2635-0d1d-49f3-bc35-eefe76a809b6/volumes"
Oct 31 01:51:04.837967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:04.837936    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" path="/var/lib/kubelet/pods/8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f/volumes"
Oct 31 01:51:14.582472 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:51:14.582436    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:51:27.055273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:27.055181    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"]
Oct 31 01:51:27.104326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:27.104288    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-9d7b2c-4blmn"]
Oct 31 01:51:28.837458 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:28.837425    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b22bd41-0191-4f9b-ab69-7b8f7917e991" path="/var/lib/kubelet/pods/5b22bd41-0191-4f9b-ab69-7b8f7917e991/volumes"
Oct 31 01:51:34.884271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884232    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"]
Oct 31 01:51:34.884813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884799    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5b22bd41-0191-4f9b-ab69-7b8f7917e991" containerName="revoke-keys"
Oct 31 01:51:34.884860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884816    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b22bd41-0191-4f9b-ab69-7b8f7917e991" containerName="revoke-keys"
Oct 31 01:51:34.884860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884859    5419 cpu_manager.go:414] "containerMap: removing container" podUID="8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" containerName="payload-pre-processing"
Oct 31 01:51:34.884928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884864    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" containerName="payload-pre-processing"
Oct 31 01:51:34.884928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884885    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2d2c2635-0d1d-49f3-bc35-eefe76a809b6" containerName="payload-processing"
Oct 31 01:51:34.884928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884890    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2c2635-0d1d-49f3-bc35-eefe76a809b6" containerName="payload-processing"
Oct 31 01:51:34.885023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884968    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b22bd41-0191-4f9b-ab69-7b8f7917e991" containerName="revoke-keys"
Oct 31 01:51:34.885023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884989    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf8ff80-a9b9-4e70-bcbf-5678e966cc7f" containerName="payload-pre-processing"
Oct 31 01:51:34.885023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.884999    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2c2635-0d1d-49f3-bc35-eefe76a809b6" containerName="payload-processing"
Oct 31 01:51:34.889242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.889224    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:34.941555 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.941516    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:34.944637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.944606    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"]
Oct 31 01:51:34.990363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.990330    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:34.990363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.990354    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:51:34.990363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:34.990354    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:35.004094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.004060    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.004291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.004190    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.004291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.004240    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsb5\" (UniqueName: \"kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.105221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.105180    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.105430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.105252    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsb5\" (UniqueName: \"kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.105430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.105409    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.105856 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.105837    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.108009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.107990    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.136445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.136353    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:51:35.161802 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.161768    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsb5\" (UniqueName: \"kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5\") pod \"maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.202841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.202807    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:35.576488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:35.576466    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"]
Oct 31 01:51:35.578783 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:51:35.578666    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a84e4ef_c6b4_4350_ab53_a9ba0561e465.slice/crio-4eeff77bef61b04f3aa548ae137e83ef3cb67d337c1804b3fdef7aa8e02aa2b5 WatchSource:0}: Error finding container 4eeff77bef61b04f3aa548ae137e83ef3cb67d337c1804b3fdef7aa8e02aa2b5: Status 404 returned error can't find the container with id 4eeff77bef61b04f3aa548ae137e83ef3cb67d337c1804b3fdef7aa8e02aa2b5
Oct 31 01:51:36.023187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:36.023124    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq" event={"ID":"3a84e4ef-c6b4-4350-ab53-a9ba0561e465","Type":"ContainerStarted","Data":"307cd886cb6cef1f17683d8f8cc5ccfa45d4ae1d2a270b673a1aca17519eeffc"}
Oct 31 01:51:36.023682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:36.023211    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq" event={"ID":"3a84e4ef-c6b4-4350-ab53-a9ba0561e465","Type":"ContainerStarted","Data":"4eeff77bef61b04f3aa548ae137e83ef3cb67d337c1804b3fdef7aa8e02aa2b5"}
Oct 31 01:51:36.137516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:36.137418    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq" podStartSLOduration=2.137404014 podStartE2EDuration="2.137404014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:36.136087032 +0000 UTC m=+2817.437043601" watchObservedRunningTime="2026-10-31 01:51:36.137404014 +0000 UTC m=+2817.438360590"
Oct 31 01:51:37.031578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:37.031546    5419 generic.go:356] "Generic (PLEG): container finished" podID="3a84e4ef-c6b4-4350-ab53-a9ba0561e465" containerID="307cd886cb6cef1f17683d8f8cc5ccfa45d4ae1d2a270b673a1aca17519eeffc" exitCode=0
Oct 31 01:51:37.031988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:37.031634    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq" event={"ID":"3a84e4ef-c6b4-4350-ab53-a9ba0561e465","Type":"ContainerDied","Data":"307cd886cb6cef1f17683d8f8cc5ccfa45d4ae1d2a270b673a1aca17519eeffc"}
Oct 31 01:51:38.185754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.185728    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:38.243537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.243501    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca\") pod \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") "
Oct 31 01:51:38.243715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.243626    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5\" (UniqueName: \"kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5\") pod \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") "
Oct 31 01:51:38.243764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.243725    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca\") pod \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\" (UID: \"3a84e4ef-c6b4-4350-ab53-a9ba0561e465\") "
Oct 31 01:51:38.243926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.243906    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca" pod "3a84e4ef-c6b4-4350-ab53-a9ba0561e465" (UID: "3a84e4ef-c6b4-4350-ab53-a9ba0561e465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:38.244177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.244128    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:38.245865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.245838    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca" pod "3a84e4ef-c6b4-4350-ab53-a9ba0561e465" (UID: "3a84e4ef-c6b4-4350-ab53-a9ba0561e465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:38.245865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.245842    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5" pod "3a84e4ef-c6b4-4350-ab53-a9ba0561e465" (UID: "3a84e4ef-c6b4-4350-ab53-a9ba0561e465"). InnerVolumeSpecName "kube-api-access-bhsb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:38.345983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.345895    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:38.345983 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:38.345930    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhsb5\" (UniqueName: \"kubernetes.io/projected/3a84e4ef-c6b4-4350-ab53-a9ba0561e465-kube-api-access-bhsb5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:39.044829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:39.044802    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"
Oct 31 01:51:39.044998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:39.044874    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq" event={"ID":"3a84e4ef-c6b4-4350-ab53-a9ba0561e465","Type":"ContainerDied","Data":"4eeff77bef61b04f3aa548ae137e83ef3cb67d337c1804b3fdef7aa8e02aa2b5"}
Oct 31 01:51:39.044998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:39.044915    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eeff77bef61b04f3aa548ae137e83ef3cb67d337c1804b3fdef7aa8e02aa2b5"
Oct 31 01:51:43.391283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.391249    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"]
Oct 31 01:51:43.440268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.440239    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-e59f12-clwkt"]
Oct 31 01:51:43.777121 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.777087    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"]
Oct 31 01:51:43.777627 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.777613    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3a84e4ef-c6b4-4350-ab53-a9ba0561e465" containerName="revoke-keys"
Oct 31 01:51:43.777678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.777629    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a84e4ef-c6b4-4350-ab53-a9ba0561e465" containerName="revoke-keys"
Oct 31 01:51:43.777757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.777748    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a84e4ef-c6b4-4350-ab53-a9ba0561e465" containerName="revoke-keys"
Oct 31 01:51:43.782595 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.782580    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:43.837954 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.837916    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"]
Oct 31 01:51:43.883927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.883888    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:43.883927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.883904    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:51:43.884174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.883974    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:43.884174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.884055    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:43.938358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.938332    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:43.938512 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.938457    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:43.938571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:43.938513    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ftz\" (UniqueName: \"kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.039400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.039324    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.039564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.039402    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ftz\" (UniqueName: \"kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.039564 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.039508    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.040050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.040032    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.042007 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.041980    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.096227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.096184    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ftz\" (UniqueName: \"kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz\") pod \"maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.096986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.096969    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:44.470780 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.470758    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"]
Oct 31 01:51:44.472553 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:51:44.472524    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2972352_5f85_4b5a_bb78_ff90cd37ec62.slice/crio-276bd894585196d42146d6abcf4a32323dd84a5bac5b09755d3701284dc23f2a WatchSource:0}: Error finding container 276bd894585196d42146d6abcf4a32323dd84a5bac5b09755d3701284dc23f2a: Status 404 returned error can't find the container with id 276bd894585196d42146d6abcf4a32323dd84a5bac5b09755d3701284dc23f2a
Oct 31 01:51:44.583613 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:51:44.583576    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:51:44.837959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:44.837930    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2c55e8-afe5-4e6b-8f70-afacc0a1c440" path="/var/lib/kubelet/pods/0c2c55e8-afe5-4e6b-8f70-afacc0a1c440/volumes"
Oct 31 01:51:45.083847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:45.083759    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm" event={"ID":"e2972352-5f85-4b5a-bb78-ff90cd37ec62","Type":"ContainerStarted","Data":"73bffe3bdfa53c75154c5c35356b7695566b14d2299f0c96bdf88b21fc36dd67"}
Oct 31 01:51:45.083847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:45.083809    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm" event={"ID":"e2972352-5f85-4b5a-bb78-ff90cd37ec62","Type":"ContainerStarted","Data":"276bd894585196d42146d6abcf4a32323dd84a5bac5b09755d3701284dc23f2a"}
Oct 31 01:51:45.196622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:45.196576    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm" podStartSLOduration=2.196557948 podStartE2EDuration="2.196557948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:45.195478503 +0000 UTC m=+2826.496435080" watchObservedRunningTime="2026-10-31 01:51:45.196557948 +0000 UTC m=+2826.497514526"
Oct 31 01:51:46.101117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:46.101076    5419 generic.go:356] "Generic (PLEG): container finished" podID="e2972352-5f85-4b5a-bb78-ff90cd37ec62" containerID="73bffe3bdfa53c75154c5c35356b7695566b14d2299f0c96bdf88b21fc36dd67" exitCode=0
Oct 31 01:51:46.101592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:46.101184    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm" event={"ID":"e2972352-5f85-4b5a-bb78-ff90cd37ec62","Type":"ContainerDied","Data":"73bffe3bdfa53c75154c5c35356b7695566b14d2299f0c96bdf88b21fc36dd67"}
Oct 31 01:51:47.270341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.270313    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:47.389617 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.389508    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca\") pod \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") "
Oct 31 01:51:47.390027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.390003    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz\" (UniqueName: \"kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz\") pod \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") "
Oct 31 01:51:47.390702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.390679    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca\") pod \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\" (UID: \"e2972352-5f85-4b5a-bb78-ff90cd37ec62\") "
Oct 31 01:51:47.395149 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.395103    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz" pod "e2972352-5f85-4b5a-bb78-ff90cd37ec62" (UID: "e2972352-5f85-4b5a-bb78-ff90cd37ec62"). InnerVolumeSpecName "kube-api-access-k9ftz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:47.396326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.396297    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca" pod "e2972352-5f85-4b5a-bb78-ff90cd37ec62" (UID: "e2972352-5f85-4b5a-bb78-ff90cd37ec62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:47.396776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.396744    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca" pod "e2972352-5f85-4b5a-bb78-ff90cd37ec62" (UID: "e2972352-5f85-4b5a-bb78-ff90cd37ec62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:51:47.492260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.492229    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ftz\" (UniqueName: \"kubernetes.io/projected/e2972352-5f85-4b5a-bb78-ff90cd37ec62-kube-api-access-k9ftz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.492260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.492257    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:47.492260 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:47.492266    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2972352-5f85-4b5a-bb78-ff90cd37ec62-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:48.116657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:48.116623    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm" event={"ID":"e2972352-5f85-4b5a-bb78-ff90cd37ec62","Type":"ContainerDied","Data":"276bd894585196d42146d6abcf4a32323dd84a5bac5b09755d3701284dc23f2a"}
Oct 31 01:51:48.116836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:48.116681    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="276bd894585196d42146d6abcf4a32323dd84a5bac5b09755d3701284dc23f2a"
Oct 31 01:51:48.116836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:48.116628    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"
Oct 31 01:51:49.843934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:49.843905    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"]
Oct 31 01:51:49.844417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:49.844128    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" containerID="cri-o://f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822" gracePeriod=30
Oct 31 01:51:54.915413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:54.915375    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 31 01:51:54.915869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:54.915447    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 31 01:51:55.106301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.106275    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:51:55.167610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.167524    5419 generic.go:356] "Generic (PLEG): container finished" podID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerID="f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822" exitCode=0
Oct 31 01:51:55.167610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.167575    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"
Oct 31 01:51:55.167829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.167603    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" event={"ID":"fc13699e-3ac8-4b5c-a459-d06cbd49c68b","Type":"ContainerDied","Data":"f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822"}
Oct 31 01:51:55.167829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.167655    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr" event={"ID":"fc13699e-3ac8-4b5c-a459-d06cbd49c68b","Type":"ContainerDied","Data":"99594a2297c30292ea90e0193ebc47e0503d0ad5b1d19d1bfa3bfb3e3c83e6e7"}
Oct 31 01:51:55.167829 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.167685    5419 scope.go:122] "RemoveContainer" containerID="f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822"
Oct 31 01:51:55.177565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.177541    5419 scope.go:122] "RemoveContainer" containerID="3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"
Oct 31 01:51:55.186559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.186545    5419 scope.go:122] "RemoveContainer" containerID="f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822"
Oct 31 01:51:55.186788 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:51:55.186770    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822\": container with ID starting with f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822 not found: ID does not exist" containerID="f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822"
Oct 31 01:51:55.186845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.186799    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822"} err="failed to get container status \"f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822\": rpc error: code = NotFound desc = could not find container \"f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822\": container with ID starting with f8c8f3bcdfbff563953b1c796d5d4032a631543c67015ed0c599640fed286822 not found: ID does not exist"
Oct 31 01:51:55.186845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.186815    5419 scope.go:122] "RemoveContainer" containerID="3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"
Oct 31 01:51:55.187059 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:51:55.187040    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139\": container with ID starting with 3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139 not found: ID does not exist" containerID="3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"
Oct 31 01:51:55.187115 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.187071    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139"} err="failed to get container status \"3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139\": rpc error: code = NotFound desc = could not find container \"3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139\": container with ID starting with 3838ff16eff19c709a6a0304712aa1cee4964e5acd34b142b8c79e9e70fc3139 not found: ID does not exist"
Oct 31 01:51:55.200459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200440    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200520    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200550    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200623 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200600    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200661 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200621    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200663    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200695    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200705    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:55.200816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200775    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200812    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket\") pod \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\" (UID: \"fc13699e-3ac8-4b5c-a459-d06cbd49c68b\") "
Oct 31 01:51:55.200925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200902    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:51:55.200925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.200907    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:55.201120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201094    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:55.201292 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201270    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:55.201633 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201615    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.201709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201639    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.201709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201654    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.201709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201668    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.201709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.201683    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.202749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.202728    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:51:55.203057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.203033    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "kube-api-access-4qbcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:55.203057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.203041    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:51:55.203203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.203065    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo" pod "fc13699e-3ac8-4b5c-a459-d06cbd49c68b" (UID: "fc13699e-3ac8-4b5c-a459-d06cbd49c68b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:51:55.303025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.302991    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.303025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.303023    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.303223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.303034    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qbcf\" (UniqueName: \"kubernetes.io/projected/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-kube-api-access-4qbcf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.303223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.303044    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc13699e-3ac8-4b5c-a459-d06cbd49c68b-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:51:55.587946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.587918    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"]
Oct 31 01:51:55.591064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:55.591041    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e59f12-openshift-default-64b5cd6757-6vxvr"]
Oct 31 01:51:56.837445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:56.837412    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" path="/var/lib/kubelet/pods/fc13699e-3ac8-4b5c-a459-d06cbd49c68b/volumes"
Oct 31 01:51:57.845034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.844995    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"]
Oct 31 01:51:57.845884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.845862    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy"
Oct 31 01:51:57.845940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.845890    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy"
Oct 31 01:51:57.845940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.845919    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e2972352-5f85-4b5a-bb78-ff90cd37ec62" containerName="revoke-keys"
Oct 31 01:51:57.845940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.845927    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2972352-5f85-4b5a-bb78-ff90cd37ec62" containerName="revoke-keys"
Oct 31 01:51:57.846114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.846100    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy"
Oct 31 01:51:57.846178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.846128    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy"
Oct 31 01:51:57.846178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.846159    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2972352-5f85-4b5a-bb78-ff90cd37ec62" containerName="revoke-keys"
Oct 31 01:51:57.851455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.851430    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:57.905339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.905305    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"]
Oct 31 01:51:57.932938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.932913    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:57.933114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.932945    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:57.933114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.932967    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5x6h\" (UniqueName: \"kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:57.951308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.951284    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:51:57.951308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.951298    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:51:57.951504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.951349    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:51:57.951504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:57.951355    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:51:58.034314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.034281    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.034314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.034318    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5x6h\" (UniqueName: \"kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.034550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.034470    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.034922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.034902    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.036941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.036922    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.093070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.093043    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5x6h\" (UniqueName: \"kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h\") pod \"maas-api-revoke-sub-2a62a293-bfq84\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.166828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.166769    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:51:58.545245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:58.545215    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"]
Oct 31 01:51:58.547224 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:51:58.547197    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7543ab5e_eaa7_4ade_ad23_c0e387518f3b.slice/crio-16383c8a2242eb2ec89603f6b23f24c0114fcd18354cb2a2fc55bfeb7e62419b WatchSource:0}: Error finding container 16383c8a2242eb2ec89603f6b23f24c0114fcd18354cb2a2fc55bfeb7e62419b: Status 404 returned error can't find the container with id 16383c8a2242eb2ec89603f6b23f24c0114fcd18354cb2a2fc55bfeb7e62419b
Oct 31 01:51:59.193390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:59.193300    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84" event={"ID":"7543ab5e-eaa7-4ade-ad23-c0e387518f3b","Type":"ContainerStarted","Data":"b0e24bc6cb7a74236ad4aa0198e204c4c2beb040e22427af99a5482a60b00c69"}
Oct 31 01:51:59.193390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:59.193366    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84" event={"ID":"7543ab5e-eaa7-4ade-ad23-c0e387518f3b","Type":"ContainerStarted","Data":"16383c8a2242eb2ec89603f6b23f24c0114fcd18354cb2a2fc55bfeb7e62419b"}
Oct 31 01:51:59.308237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:51:59.308169    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84" podStartSLOduration=2.308132929 podStartE2EDuration="2.308132929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:59.305305487 +0000 UTC m=+2840.606262064" watchObservedRunningTime="2026-10-31 01:51:59.308132929 +0000 UTC m=+2840.609089507"
Oct 31 01:52:00.034445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:00.034408    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"]
Oct 31 01:52:00.036875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:00.036854    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-99145d-kn49j"]
Oct 31 01:52:00.200898 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:00.200867    5419 generic.go:356] "Generic (PLEG): container finished" podID="7543ab5e-eaa7-4ade-ad23-c0e387518f3b" containerID="b0e24bc6cb7a74236ad4aa0198e204c4c2beb040e22427af99a5482a60b00c69" exitCode=0
Oct 31 01:52:00.201302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:00.200959    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84" event={"ID":"7543ab5e-eaa7-4ade-ad23-c0e387518f3b","Type":"ContainerDied","Data":"b0e24bc6cb7a74236ad4aa0198e204c4c2beb040e22427af99a5482a60b00c69"}
Oct 31 01:52:00.837368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:00.837342    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ecaf011-0207-42ef-854f-ef6caf47b699" path="/var/lib/kubelet/pods/4ecaf011-0207-42ef-854f-ef6caf47b699/volumes"
Oct 31 01:52:01.354744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.354715    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:52:01.478034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.478004    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca\") pod \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") "
Oct 31 01:52:01.478247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.478126    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h\" (UniqueName: \"kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h\") pod \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") "
Oct 31 01:52:01.478247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.478180    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca\") pod \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\" (UID: \"7543ab5e-eaa7-4ade-ad23-c0e387518f3b\") "
Oct 31 01:52:01.478695 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.478662    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca" pod "7543ab5e-eaa7-4ade-ad23-c0e387518f3b" (UID: "7543ab5e-eaa7-4ade-ad23-c0e387518f3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:01.484277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.484253    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca" pod "7543ab5e-eaa7-4ade-ad23-c0e387518f3b" (UID: "7543ab5e-eaa7-4ade-ad23-c0e387518f3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:01.484423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.484288    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h" pod "7543ab5e-eaa7-4ade-ad23-c0e387518f3b" (UID: "7543ab5e-eaa7-4ade-ad23-c0e387518f3b"). InnerVolumeSpecName "kube-api-access-x5x6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:01.579551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.579523    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:01.579551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.579549    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5x6h\" (UniqueName: \"kubernetes.io/projected/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-kube-api-access-x5x6h\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:01.579774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:01.579560    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7543ab5e-eaa7-4ade-ad23-c0e387518f3b-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:02.213962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:02.213919    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84" event={"ID":"7543ab5e-eaa7-4ade-ad23-c0e387518f3b","Type":"ContainerDied","Data":"16383c8a2242eb2ec89603f6b23f24c0114fcd18354cb2a2fc55bfeb7e62419b"}
Oct 31 01:52:02.214201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:02.213985    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16383c8a2242eb2ec89603f6b23f24c0114fcd18354cb2a2fc55bfeb7e62419b"
Oct 31 01:52:02.214201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:02.213947    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"
Oct 31 01:52:06.654896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:06.654860    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"]
Oct 31 01:52:06.655351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:06.655071    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" containerID="cri-o://accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606" gracePeriod=30
Oct 31 01:52:07.658119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:07.658084    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:52:09.506861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:09.506832    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:52:09.507291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:09.506891    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:52:11.927824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:11.927770    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:52:12.101975 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.101938    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102021    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102094    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102133    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102196    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102232    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102296    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102339    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:12.102562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102372    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102444    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy\") pod \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\" (UID: \"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8\") "
Oct 31 01:52:12.102562 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102543    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:12.102952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102918    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.102952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.102948    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.103620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.103594    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:12.103815 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.103793    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:12.105691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.105503    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "kube-api-access-xvdzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:12.106003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.105949    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:12.106314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.106292    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:12.108499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.108476    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:12.111274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.111247    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo" pod "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" (UID: "ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.203924    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvdzg\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-kube-api-access-xvdzg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.203954    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.203964    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.203973    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.203981    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.203989    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.204000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.204001    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:12.277911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.277877    5419 generic.go:356] "Generic (PLEG): container finished" podID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerID="accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606" exitCode=0
Oct 31 01:52:12.278075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.277935    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"
Oct 31 01:52:12.278075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.277963    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" event={"ID":"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8","Type":"ContainerDied","Data":"accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606"}
Oct 31 01:52:12.278075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.278018    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq" event={"ID":"ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8","Type":"ContainerDied","Data":"e13fbaf0d5f91a1f9924ab918b1a9f5255e4a3bfd093e1c9ddd3e093d83c7bf2"}
Oct 31 01:52:12.278075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.278043    5419 scope.go:122] "RemoveContainer" containerID="accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606"
Oct 31 01:52:12.289222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.289206    5419 scope.go:122] "RemoveContainer" containerID="5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"
Oct 31 01:52:12.298441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.298424    5419 scope.go:122] "RemoveContainer" containerID="accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606"
Oct 31 01:52:12.298715 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:52:12.298696    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606\": container with ID starting with accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606 not found: ID does not exist" containerID="accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606"
Oct 31 01:52:12.298769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.298722    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606"} err="failed to get container status \"accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606\": rpc error: code = NotFound desc = could not find container \"accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606\": container with ID starting with accb166b9aab7e7bcf75f59be8d2d1ae169586f2a4419d878b1232527b6e3606 not found: ID does not exist"
Oct 31 01:52:12.298769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.298739    5419 scope.go:122] "RemoveContainer" containerID="5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"
Oct 31 01:52:12.298966 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:52:12.298947    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d\": container with ID starting with 5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d not found: ID does not exist" containerID="5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"
Oct 31 01:52:12.299009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.298974    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d"} err="failed to get container status \"5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d\": rpc error: code = NotFound desc = could not find container \"5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d\": container with ID starting with 5e4953c4d1da4dec17e070f031b53fc31768524db642849242c2945de38ef46d not found: ID does not exist"
Oct 31 01:52:12.446176 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.446122    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"]
Oct 31 01:52:12.449605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.449583    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-99145d-openshift-default-8666b49cb5-dbdhq"]
Oct 31 01:52:12.837679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:12.837629    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" path="/var/lib/kubelet/pods/ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8/volumes"
Oct 31 01:52:14.255477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.255439    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"]
Oct 31 01:52:14.255952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.255659    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm" podUID="d3da6392-a65c-483e-ae2b-35ad51a4b4db" containerName="payload-processing" containerID="cri-o://2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541" gracePeriod=30
Oct 31 01:52:14.539615 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.539590    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:52:14.585775 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:52:14.585731    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:52:14.639945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.639912    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z\" (UniqueName: \"kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z\") pod \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") "
Oct 31 01:52:14.640130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.640057    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume\") pod \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\" (UID: \"d3da6392-a65c-483e-ae2b-35ad51a4b4db\") "
Oct 31 01:52:14.640447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.640413    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume" pod "d3da6392-a65c-483e-ae2b-35ad51a4b4db" (UID: "d3da6392-a65c-483e-ae2b-35ad51a4b4db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:14.642198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.642178    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z" pod "d3da6392-a65c-483e-ae2b-35ad51a4b4db" (UID: "d3da6392-a65c-483e-ae2b-35ad51a4b4db"). InnerVolumeSpecName "kube-api-access-lhq5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:14.741188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.741157    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhq5z\" (UniqueName: \"kubernetes.io/projected/d3da6392-a65c-483e-ae2b-35ad51a4b4db-kube-api-access-lhq5z\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:14.741188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:14.741183    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3da6392-a65c-483e-ae2b-35ad51a4b4db-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:15.298489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.298452    5419 generic.go:356] "Generic (PLEG): container finished" podID="d3da6392-a65c-483e-ae2b-35ad51a4b4db" containerID="2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541" exitCode=0
Oct 31 01:52:15.298919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.298500    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"
Oct 31 01:52:15.298919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.298540    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm" event={"ID":"d3da6392-a65c-483e-ae2b-35ad51a4b4db","Type":"ContainerDied","Data":"2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541"}
Oct 31 01:52:15.298919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.298594    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm" event={"ID":"d3da6392-a65c-483e-ae2b-35ad51a4b4db","Type":"ContainerDied","Data":"282b3f9ce4c91fb45c46b46c87081271e04c6d654dabf3c9f5a0ee30e1cee469"}
Oct 31 01:52:15.298919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.298624    5419 scope.go:122] "RemoveContainer" containerID="2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541"
Oct 31 01:52:15.308519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.308494    5419 scope.go:122] "RemoveContainer" containerID="2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541"
Oct 31 01:52:15.308778 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:52:15.308760    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541\": container with ID starting with 2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541 not found: ID does not exist" containerID="2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541"
Oct 31 01:52:15.308826 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.308787    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541"} err="failed to get container status \"2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541\": rpc error: code = NotFound desc = could not find container \"2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541\": container with ID starting with 2d9edf6dd8cd8c8373146e77c69788c3db7dff2e96ceb7fc38831335aba61541 not found: ID does not exist"
Oct 31 01:52:15.462703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.462671    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"]
Oct 31 01:52:15.511577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:15.511545    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b2d49b-7d964db466-npfgm"]
Oct 31 01:52:16.026415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:16.026385    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"]
Oct 31 01:52:16.031310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:16.031286    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b2d49b-ftbjm"]
Oct 31 01:52:16.837000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:16.836970    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3da6392-a65c-483e-ae2b-35ad51a4b4db" path="/var/lib/kubelet/pods/d3da6392-a65c-483e-ae2b-35ad51a4b4db/volumes"
Oct 31 01:52:16.837447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:16.837432    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2972352-5f85-4b5a-bb78-ff90cd37ec62" path="/var/lib/kubelet/pods/e2972352-5f85-4b5a-bb78-ff90cd37ec62/volumes"
Oct 31 01:52:22.358929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.358896    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"]
Oct 31 01:52:22.359486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359470    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy"
Oct 31 01:52:22.359536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359488    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy"
Oct 31 01:52:22.359536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359513    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy"
Oct 31 01:52:22.359536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359519    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy"
Oct 31 01:52:22.359536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359533    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359538    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc13699e-3ac8-4b5c-a459-d06cbd49c68b" containerName="istio-proxy"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359562    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d3da6392-a65c-483e-ae2b-35ad51a4b4db" containerName="payload-processing"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359568    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3da6392-a65c-483e-ae2b-35ad51a4b4db" containerName="payload-processing"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359574    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7543ab5e-eaa7-4ade-ad23-c0e387518f3b" containerName="revoke-keys"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359578    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7543ab5e-eaa7-4ade-ad23-c0e387518f3b" containerName="revoke-keys"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359668    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359680    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3da6392-a65c-483e-ae2b-35ad51a4b4db" containerName="payload-processing"
Oct 31 01:52:22.359683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359686    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7543ab5e-eaa7-4ade-ad23-c0e387518f3b" containerName="revoke-keys"
Oct 31 01:52:22.359930 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.359695    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccffd2b0-3dcb-42b3-adb1-3c57482d6ea8" containerName="istio-proxy"
Oct 31 01:52:22.364357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.364339    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.419853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.419815    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"]
Oct 31 01:52:22.434822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.434785    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrhl\" (UniqueName: \"kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.434987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.434918    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.435073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.435056    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.465483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.465457    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:22.465602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.465480    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:22.465602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.465474    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:22.465602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.465510    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:52:22.536692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.536656    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.536843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.536756    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.536843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.536799    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrhl\" (UniqueName: \"kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.537489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.537471    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.539249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.539227    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.593274 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.593246    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrhl\" (UniqueName: \"kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl\") pod \"maas-api-revoke-sub-5b6b4161-x77f8\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:22.677730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:22.677666    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:23.050538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:23.050513    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"]
Oct 31 01:52:23.052355 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:23.052326    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d4138c_e763_4909_803f_78068715bc55.slice/crio-d6079d97167ebf59d52106f23500ce8da008173c5b21cb767e655126bba9bcf3 WatchSource:0}: Error finding container d6079d97167ebf59d52106f23500ce8da008173c5b21cb767e655126bba9bcf3: Status 404 returned error can't find the container with id d6079d97167ebf59d52106f23500ce8da008173c5b21cb767e655126bba9bcf3
Oct 31 01:52:23.361120 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:23.361024    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8" event={"ID":"f5d4138c-e763-4909-803f-78068715bc55","Type":"ContainerStarted","Data":"d6079d97167ebf59d52106f23500ce8da008173c5b21cb767e655126bba9bcf3"}
Oct 31 01:52:24.368632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:24.368592    5419 generic.go:356] "Generic (PLEG): container finished" podID="f5d4138c-e763-4909-803f-78068715bc55" containerID="e1ffd0522a26427068c7a4e2a0374690cffc74d846ead12bcce419b5a94e641f" exitCode=0
Oct 31 01:52:24.369102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:24.368677    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8" event={"ID":"f5d4138c-e763-4909-803f-78068715bc55","Type":"ContainerDied","Data":"e1ffd0522a26427068c7a4e2a0374690cffc74d846ead12bcce419b5a94e641f"}
Oct 31 01:52:25.517970 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.517946    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:25.572488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.572462    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca\") pod \"f5d4138c-e763-4909-803f-78068715bc55\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") "
Oct 31 01:52:25.572622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.572507    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl\" (UniqueName: \"kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl\") pod \"f5d4138c-e763-4909-803f-78068715bc55\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") "
Oct 31 01:52:25.572622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.572616    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca\") pod \"f5d4138c-e763-4909-803f-78068715bc55\" (UID: \"f5d4138c-e763-4909-803f-78068715bc55\") "
Oct 31 01:52:25.572790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.572769    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca" pod "f5d4138c-e763-4909-803f-78068715bc55" (UID: "f5d4138c-e763-4909-803f-78068715bc55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:25.572962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.572943    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d4138c-e763-4909-803f-78068715bc55-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:25.574755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.574732    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca" pod "f5d4138c-e763-4909-803f-78068715bc55" (UID: "f5d4138c-e763-4909-803f-78068715bc55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:25.574855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.574817    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl" pod "f5d4138c-e763-4909-803f-78068715bc55" (UID: "f5d4138c-e763-4909-803f-78068715bc55"). InnerVolumeSpecName "kube-api-access-vqrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:25.673979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.673894    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrhl\" (UniqueName: \"kubernetes.io/projected/f5d4138c-e763-4909-803f-78068715bc55-kube-api-access-vqrhl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:25.673979 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:25.673930    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d4138c-e763-4909-803f-78068715bc55-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:26.281432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.281393    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"]
Oct 31 01:52:26.281876 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.281864    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f5d4138c-e763-4909-803f-78068715bc55" containerName="revoke-keys"
Oct 31 01:52:26.281924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.281877    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d4138c-e763-4909-803f-78068715bc55" containerName="revoke-keys"
Oct 31 01:52:26.281997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.281988    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d4138c-e763-4909-803f-78068715bc55" containerName="revoke-keys"
Oct 31 01:52:26.285418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.285402    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.342590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.342557    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"]
Oct 31 01:52:26.382517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.382489    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.382683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.382638    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.382746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.382710    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbbf\" (UniqueName: \"kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.383419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.383404    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"
Oct 31 01:52:26.383482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.383420    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8" event={"ID":"f5d4138c-e763-4909-803f-78068715bc55","Type":"ContainerDied","Data":"d6079d97167ebf59d52106f23500ce8da008173c5b21cb767e655126bba9bcf3"}
Oct 31 01:52:26.383482 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.383458    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6079d97167ebf59d52106f23500ce8da008173c5b21cb767e655126bba9bcf3"
Oct 31 01:52:26.484374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.484343    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.484374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.484378    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbbf\" (UniqueName: \"kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.484607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.484513    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.485089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.485067    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.487181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.487158    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.545759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.545691    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbbf\" (UniqueName: \"kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf\") pod \"maas-api-revoke-sub-feb4ec3a-97vj2\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.598557 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.598529    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:26.972635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:26.972609    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"]
Oct 31 01:52:26.974374 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:26.974347    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0537df70_0add_4011_98a4_9bc3480b5933.slice/crio-266d13aee897c60f01c7b97e2abb5456384bcdb3c0ac5decc8568b9d16e4523d WatchSource:0}: Error finding container 266d13aee897c60f01c7b97e2abb5456384bcdb3c0ac5decc8568b9d16e4523d: Status 404 returned error can't find the container with id 266d13aee897c60f01c7b97e2abb5456384bcdb3c0ac5decc8568b9d16e4523d
Oct 31 01:52:27.390890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:27.390847    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2" event={"ID":"0537df70-0add-4011-98a4-9bc3480b5933","Type":"ContainerStarted","Data":"4c9d0735a39b062d4a37dd7597a2300a7b06e967e89a3e4e614df9d93bda3c73"}
Oct 31 01:52:27.391066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:27.390931    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2" event={"ID":"0537df70-0add-4011-98a4-9bc3480b5933","Type":"ContainerStarted","Data":"266d13aee897c60f01c7b97e2abb5456384bcdb3c0ac5decc8568b9d16e4523d"}
Oct 31 01:52:27.505123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:27.505070    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2" podStartSLOduration=1.5050562269999999 podStartE2EDuration="1.505056227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:27.503295723 +0000 UTC m=+2868.804252305" watchObservedRunningTime="2026-10-31 01:52:27.505056227 +0000 UTC m=+2868.806012804"
Oct 31 01:52:28.397994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:28.397962    5419 generic.go:356] "Generic (PLEG): container finished" podID="0537df70-0add-4011-98a4-9bc3480b5933" containerID="4c9d0735a39b062d4a37dd7597a2300a7b06e967e89a3e4e614df9d93bda3c73" exitCode=0
Oct 31 01:52:28.398451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:28.398054    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2" event={"ID":"0537df70-0add-4011-98a4-9bc3480b5933","Type":"ContainerDied","Data":"4c9d0735a39b062d4a37dd7597a2300a7b06e967e89a3e4e614df9d93bda3c73"}
Oct 31 01:52:29.545889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.545856    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:29.621021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.620994    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf\" (UniqueName: \"kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf\") pod \"0537df70-0add-4011-98a4-9bc3480b5933\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") "
Oct 31 01:52:29.621212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.621200    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca\") pod \"0537df70-0add-4011-98a4-9bc3480b5933\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") "
Oct 31 01:52:29.621264 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.621234    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca\") pod \"0537df70-0add-4011-98a4-9bc3480b5933\" (UID: \"0537df70-0add-4011-98a4-9bc3480b5933\") "
Oct 31 01:52:29.621545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.621523    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca" pod "0537df70-0add-4011-98a4-9bc3480b5933" (UID: "0537df70-0add-4011-98a4-9bc3480b5933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:29.623247 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.623226    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf" pod "0537df70-0add-4011-98a4-9bc3480b5933" (UID: "0537df70-0add-4011-98a4-9bc3480b5933"). InnerVolumeSpecName "kube-api-access-xhbbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:29.623372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.623338    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca" pod "0537df70-0add-4011-98a4-9bc3480b5933" (UID: "0537df70-0add-4011-98a4-9bc3480b5933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:29.722448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.722417    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:29.722448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.722446    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0537df70-0add-4011-98a4-9bc3480b5933-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:29.722646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:29.722457    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbbf\" (UniqueName: \"kubernetes.io/projected/0537df70-0add-4011-98a4-9bc3480b5933-kube-api-access-xhbbf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:30.411338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:30.411308    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"
Oct 31 01:52:30.411517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:30.411308    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2" event={"ID":"0537df70-0add-4011-98a4-9bc3480b5933","Type":"ContainerDied","Data":"266d13aee897c60f01c7b97e2abb5456384bcdb3c0ac5decc8568b9d16e4523d"}
Oct 31 01:52:30.411517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:30.411431    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="266d13aee897c60f01c7b97e2abb5456384bcdb3c0ac5decc8568b9d16e4523d"
Oct 31 01:52:34.685375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.685336    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"]
Oct 31 01:52:34.686068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.686046    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0537df70-0add-4011-98a4-9bc3480b5933" containerName="revoke-keys"
Oct 31 01:52:34.686191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.686070    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0537df70-0add-4011-98a4-9bc3480b5933" containerName="revoke-keys"
Oct 31 01:52:34.686296 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.686282    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0537df70-0add-4011-98a4-9bc3480b5933" containerName="revoke-keys"
Oct 31 01:52:34.689956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.689933    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.742651 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.742624    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-3b7279-openshift-default-dockercfg-gwsz5\""
Oct 31 01:52:34.783530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.783216    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.783866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.783839    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784013    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784181    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784322    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmbf5\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784453    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784614    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784803    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.784922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.784906    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.821357 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.801549    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"]
Oct 31 01:52:34.893922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.893881    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.895094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.894574    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.895644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.895626    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.896213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.896193    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.897209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.897158    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.897445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.897427    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.897678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.897663    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.898270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.898252    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.898836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.898817    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.899464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.899434    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmbf5\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.899722 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.899695    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.900045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.900018    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.900128 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.898675    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.900338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.900322    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.902625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.902601    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.902999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.902977    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.954971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.954894    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:34.956311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:34.956282    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmbf5\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5\") pod \"e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") " pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:35.003022 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:35.002990    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:35.431788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:35.431757    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"]
Oct 31 01:52:35.433790 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:35.433755    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a9a077_da45_490a_8148_6069b2f37d0c.slice/crio-0e472ea691f0b2193735e54612b6fdcce7dad3e53a276507a57266559dec915c WatchSource:0}: Error finding container 0e472ea691f0b2193735e54612b6fdcce7dad3e53a276507a57266559dec915c: Status 404 returned error can't find the container with id 0e472ea691f0b2193735e54612b6fdcce7dad3e53a276507a57266559dec915c
Oct 31 01:52:35.436513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:35.436481    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:52:35.436624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:35.436562    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:52:35.436624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:35.436607    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:52:36.448893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:36.448854    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" event={"ID":"f9a9a077-da45-490a-8148-6069b2f37d0c","Type":"ContainerStarted","Data":"b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"}
Oct 31 01:52:36.449352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:36.448915    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" event={"ID":"f9a9a077-da45-490a-8148-6069b2f37d0c","Type":"ContainerStarted","Data":"0e472ea691f0b2193735e54612b6fdcce7dad3e53a276507a57266559dec915c"}
Oct 31 01:52:36.616008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:36.615944    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podStartSLOduration=2.615930693 podStartE2EDuration="2.615930693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:36.613830984 +0000 UTC m=+2877.914787561" watchObservedRunningTime="2026-10-31 01:52:36.615930693 +0000 UTC m=+2877.916887269"
Oct 31 01:52:37.003355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:37.003323    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:37.009021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:37.008998    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:37.455783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:37.455708    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:37.456679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:37.456660    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:52:37.590374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:37.590344    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:52:40.290001 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.289959    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"]
Oct 31 01:52:40.294504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.294484    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.384835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.384798    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.385018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.384841    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.385018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.384970    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64xql\" (UniqueName: \"kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.395995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.395963    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:52:40.395995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.395989    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:40.396210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.395970    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:40.396210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.395965    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:40.398931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.398904    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"]
Oct 31 01:52:40.485978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.485945    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.486195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.485995    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.486195 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.486038    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64xql\" (UniqueName: \"kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.486891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.486865    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.488701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.488673    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.546597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.546516    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64xql\" (UniqueName: \"kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql\") pod \"maas-api-revoke-sub-f62615d3-xlnbd\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:40.608910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:40.608877    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:41.000404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:41.000375    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"]
Oct 31 01:52:41.002069 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:41.002031    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c24cdc9_ecdb_49fb_b5be_bf4ac41308d8.slice/crio-bd42c888cc14744f6751bb506e580cde74798384326d22a7335fac59a63130b5 WatchSource:0}: Error finding container bd42c888cc14744f6751bb506e580cde74798384326d22a7335fac59a63130b5: Status 404 returned error can't find the container with id bd42c888cc14744f6751bb506e580cde74798384326d22a7335fac59a63130b5
Oct 31 01:52:41.484486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:41.484449    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd" event={"ID":"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8","Type":"ContainerStarted","Data":"0c4b539b064370f160e2de7bf2f1428ebe2edf9b8126964ba9f2ee05bfe59ccc"}
Oct 31 01:52:41.484902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:41.484512    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd" event={"ID":"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8","Type":"ContainerStarted","Data":"bd42c888cc14744f6751bb506e580cde74798384326d22a7335fac59a63130b5"}
Oct 31 01:52:41.603917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:41.603861    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd" podStartSLOduration=1.603846842 podStartE2EDuration="1.603846842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:41.601939509 +0000 UTC m=+2882.902896087" watchObservedRunningTime="2026-10-31 01:52:41.603846842 +0000 UTC m=+2882.904803419"
Oct 31 01:52:42.492488 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.492457    5419 generic.go:356] "Generic (PLEG): container finished" podID="5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" containerID="0c4b539b064370f160e2de7bf2f1428ebe2edf9b8126964ba9f2ee05bfe59ccc" exitCode=0
Oct 31 01:52:42.492924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.492539    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd" event={"ID":"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8","Type":"ContainerDied","Data":"0c4b539b064370f160e2de7bf2f1428ebe2edf9b8126964ba9f2ee05bfe59ccc"}
Oct 31 01:52:42.659843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.659811    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"]
Oct 31 01:52:42.663934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.663913    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.720827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.720796    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"]
Oct 31 01:52:42.727211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.727179    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.727399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.727343    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.727399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.727391    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwt9c\" (UniqueName: \"kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.790791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.790690    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"]
Oct 31 01:52:42.793188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.793151    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2c0de9-lvqnq"]
Oct 31 01:52:42.828652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.828625    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.828820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.828687    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwt9c\" (UniqueName: \"kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.828820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.828789    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.829332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.829313    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.831092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.831067    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.838737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.838713    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a84e4ef-c6b4-4350-ab53-a9ba0561e465" path="/var/lib/kubelet/pods/3a84e4ef-c6b4-4350-ab53-a9ba0561e465/volumes"
Oct 31 01:52:42.885054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.885025    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwt9c\" (UniqueName: \"kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c\") pod \"maas-api-revoke-sub-57338433-wnjqd\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:42.978192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:42.978164    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:43.357046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.357017    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"]
Oct 31 01:52:43.358577 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:43.358545    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d5045a_a6a7_4c42_bcea_fa6b13cb7f18.slice/crio-f73468b5c0314e42e9bfa91fa8c560f6445879042d5642c3d09bb76965049018 WatchSource:0}: Error finding container f73468b5c0314e42e9bfa91fa8c560f6445879042d5642c3d09bb76965049018: Status 404 returned error can't find the container with id f73468b5c0314e42e9bfa91fa8c560f6445879042d5642c3d09bb76965049018
Oct 31 01:52:43.500586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.500544    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd" event={"ID":"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18","Type":"ContainerStarted","Data":"f73468b5c0314e42e9bfa91fa8c560f6445879042d5642c3d09bb76965049018"}
Oct 31 01:52:43.632215 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.632188    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:43.741432 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.741405    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql\" (UniqueName: \"kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql\") pod \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") "
Oct 31 01:52:43.741590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.741504    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca\") pod \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") "
Oct 31 01:52:43.741590 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.741545    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca\") pod \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\" (UID: \"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8\") "
Oct 31 01:52:43.741934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.741910    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca" pod "5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" (UID: "5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:43.743600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.743567    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql" pod "5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" (UID: "5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8"). InnerVolumeSpecName "kube-api-access-64xql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:43.743708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.743657    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca" pod "5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" (UID: "5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:43.843768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.843736    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:43.843900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.843772    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:43.843900 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:43.843787    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64xql\" (UniqueName: \"kubernetes.io/projected/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8-kube-api-access-64xql\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:44.508125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:44.508091    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"
Oct 31 01:52:44.508575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:44.508087    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd" event={"ID":"5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8","Type":"ContainerDied","Data":"bd42c888cc14744f6751bb506e580cde74798384326d22a7335fac59a63130b5"}
Oct 31 01:52:44.508575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:44.508193    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd42c888cc14744f6751bb506e580cde74798384326d22a7335fac59a63130b5"
Oct 31 01:52:44.509786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:44.509762    5419 generic.go:356] "Generic (PLEG): container finished" podID="06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" containerID="5356138bb8d7643a115b01a8dc467ee0fe4a83a595e6356a85f614f4dbd5db71" exitCode=0
Oct 31 01:52:44.509899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:44.509848    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd" event={"ID":"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18","Type":"ContainerDied","Data":"5356138bb8d7643a115b01a8dc467ee0fe4a83a595e6356a85f614f4dbd5db71"}
Oct 31 01:52:44.582175 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:52:44.582074    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:52:45.654445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.654421    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:45.765097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.765052    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca\") pod \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") "
Oct 31 01:52:45.765323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.765200    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c\" (UniqueName: \"kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c\") pod \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") "
Oct 31 01:52:45.765323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.765252    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca\") pod \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\" (UID: \"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18\") "
Oct 31 01:52:45.765454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.765399    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca" pod "06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" (UID: "06d5045a-a6a7-4c42-bcea-fa6b13cb7f18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:45.765646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.765627    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:45.768246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.768163    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c" pod "06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" (UID: "06d5045a-a6a7-4c42-bcea-fa6b13cb7f18"). InnerVolumeSpecName "kube-api-access-qwt9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:45.768382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.768303    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca" pod "06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" (UID: "06d5045a-a6a7-4c42-bcea-fa6b13cb7f18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:45.866580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.866545    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwt9c\" (UniqueName: \"kubernetes.io/projected/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-kube-api-access-qwt9c\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:45.866580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:45.866581    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:46.524345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:46.524312    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"
Oct 31 01:52:46.524345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:46.524339    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd" event={"ID":"06d5045a-a6a7-4c42-bcea-fa6b13cb7f18","Type":"ContainerDied","Data":"f73468b5c0314e42e9bfa91fa8c560f6445879042d5642c3d09bb76965049018"}
Oct 31 01:52:46.524570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:46.524388    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f73468b5c0314e42e9bfa91fa8c560f6445879042d5642c3d09bb76965049018"
Oct 31 01:52:49.356732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:49.356654    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"]
Oct 31 01:52:49.357169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:49.356866    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" containerID="cri-o://ec4a24e698089ebb6ebd34740397fcb9634b3dc96f8b94ca4d6c1a93c4e8ba33" gracePeriod=30
Oct 31 01:52:51.148062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.148022    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"]
Oct 31 01:52:51.148776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.148753    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" containerName="revoke-keys"
Oct 31 01:52:51.148891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.148778    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" containerName="revoke-keys"
Oct 31 01:52:51.148891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.148818    5419 cpu_manager.go:414] "containerMap: removing container" podUID="06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" containerName="revoke-keys"
Oct 31 01:52:51.148891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.148826    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" containerName="revoke-keys"
Oct 31 01:52:51.149061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.148993    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" containerName="revoke-keys"
Oct 31 01:52:51.149061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.149017    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" containerName="revoke-keys"
Oct 31 01:52:51.152438 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.152413    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.229489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.229457    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.229489 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.229499    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplmm\" (UniqueName: \"kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.229734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.229653    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.255030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.255003    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:51.255030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.255021    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:51.255030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.255032    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:52:51.255338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.255100    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:51.257624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.257604    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"]
Oct 31 01:52:51.331159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.331102    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.331328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.331163    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplmm\" (UniqueName: \"kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.331328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.331220    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.331755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.331735    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.333629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.333604    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.363495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.363468    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:52:51.363655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.363527    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:52:51.388212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.388182    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplmm\" (UniqueName: \"kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm\") pod \"maas-api-revoke-sub-0e583f52-gz9bl\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.466905 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.466830    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:51.845426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:51.845399    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"]
Oct 31 01:52:51.847468 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:51.847440    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a02b55_0131_4427_ac05_0c21235dde03.slice/crio-1ccea4e99fc6cbfe28e390a525c8c585c12cab33ca3a6a176054d9ca4c278799 WatchSource:0}: Error finding container 1ccea4e99fc6cbfe28e390a525c8c585c12cab33ca3a6a176054d9ca4c278799: Status 404 returned error can't find the container with id 1ccea4e99fc6cbfe28e390a525c8c585c12cab33ca3a6a176054d9ca4c278799
Oct 31 01:52:52.562470 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:52.562434    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl" event={"ID":"c4a02b55-0131-4427-ac05-0c21235dde03","Type":"ContainerStarted","Data":"45c5bc4111e0e591d75b67f608727da66655e5afbe136f811cb4508e9d6f501e"}
Oct 31 01:52:52.562870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:52.562487    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl" event={"ID":"c4a02b55-0131-4427-ac05-0c21235dde03","Type":"ContainerStarted","Data":"1ccea4e99fc6cbfe28e390a525c8c585c12cab33ca3a6a176054d9ca4c278799"}
Oct 31 01:52:52.677709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:52.677643    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl" podStartSLOduration=1.677628597 podStartE2EDuration="1.677628597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:52.675338882 +0000 UTC m=+2893.976295475" watchObservedRunningTime="2026-10-31 01:52:52.677628597 +0000 UTC m=+2893.978585173"
Oct 31 01:52:53.571380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:53.571350    5419 generic.go:356] "Generic (PLEG): container finished" podID="c4a02b55-0131-4427-ac05-0c21235dde03" containerID="45c5bc4111e0e591d75b67f608727da66655e5afbe136f811cb4508e9d6f501e" exitCode=0
Oct 31 01:52:53.571816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:53.571440    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl" event={"ID":"c4a02b55-0131-4427-ac05-0c21235dde03","Type":"ContainerDied","Data":"45c5bc4111e0e591d75b67f608727da66655e5afbe136f811cb4508e9d6f501e"}
Oct 31 01:52:54.581016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.580985    5419 generic.go:356] "Generic (PLEG): container finished" podID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerID="ec4a24e698089ebb6ebd34740397fcb9634b3dc96f8b94ca4d6c1a93c4e8ba33" exitCode=0
Oct 31 01:52:54.581511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.581056    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" event={"ID":"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97","Type":"ContainerDied","Data":"ec4a24e698089ebb6ebd34740397fcb9634b3dc96f8b94ca4d6c1a93c4e8ba33"}
Oct 31 01:52:54.581511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.581122    5419 scope.go:122] "RemoveContainer" containerID="4ff5f0d5a424e6a567972a20e9ce95d212ca78c2cb589557587025fa9644933e"
Oct 31 01:52:54.625097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.625032    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:52:54.669718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.669689    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.669890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.669742    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.669890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.669777    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.669890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.669822    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.670053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.669966    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.670053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670039    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.670187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670038    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:54.670187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670086    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.670187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670159    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.670352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670215    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy\") pod \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\" (UID: \"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97\") "
Oct 31 01:52:54.670352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670265    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:54.670450 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670428    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:54.670789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670544    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:54.670789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670655    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:54.671006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670846    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.671006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670866    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.671006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670878    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.671006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670891    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.671006 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.670917    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.673079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.673050    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "kube-api-access-c2fqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:54.673079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.673069    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:52:54.673751 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.673717    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:54.674920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.674891    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy" pod "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" (UID: "c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:52:54.720473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.720449    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:54.771774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.771743    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm\" (UniqueName: \"kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm\") pod \"c4a02b55-0131-4427-ac05-0c21235dde03\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") "
Oct 31 01:52:54.771923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.771856    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca\") pod \"c4a02b55-0131-4427-ac05-0c21235dde03\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") "
Oct 31 01:52:54.771977 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.771944    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca\") pod \"c4a02b55-0131-4427-ac05-0c21235dde03\" (UID: \"c4a02b55-0131-4427-ac05-0c21235dde03\") "
Oct 31 01:52:54.772229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.772204    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca" pod "c4a02b55-0131-4427-ac05-0c21235dde03" (UID: "c4a02b55-0131-4427-ac05-0c21235dde03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:52:54.772320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.772280    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.772320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.772301    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2fqf\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-kube-api-access-c2fqf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.772320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.772317    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.772439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.772326    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.772439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.772334    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.773984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.773962    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm" pod "c4a02b55-0131-4427-ac05-0c21235dde03" (UID: "c4a02b55-0131-4427-ac05-0c21235dde03"). InnerVolumeSpecName "kube-api-access-mplmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:52:54.774060 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.774006    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca" pod "c4a02b55-0131-4427-ac05-0c21235dde03" (UID: "c4a02b55-0131-4427-ac05-0c21235dde03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:52:54.873678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.873627    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a02b55-0131-4427-ac05-0c21235dde03-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:54.873678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:54.873657    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplmm\" (UniqueName: \"kubernetes.io/projected/c4a02b55-0131-4427-ac05-0c21235dde03-kube-api-access-mplmm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:52:55.589736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.589705    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"
Oct 31 01:52:55.590227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.589747    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv" event={"ID":"c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97","Type":"ContainerDied","Data":"b69ffd622c4b22cffefc7ecbf2eced831954e1b1d82a4dad2a7db9005928917d"}
Oct 31 01:52:55.590227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.589806    5419 scope.go:122] "RemoveContainer" containerID="ec4a24e698089ebb6ebd34740397fcb9634b3dc96f8b94ca4d6c1a93c4e8ba33"
Oct 31 01:52:55.592233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.591966    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"
Oct 31 01:52:55.592233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.591978    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl" event={"ID":"c4a02b55-0131-4427-ac05-0c21235dde03","Type":"ContainerDied","Data":"1ccea4e99fc6cbfe28e390a525c8c585c12cab33ca3a6a176054d9ca4c278799"}
Oct 31 01:52:55.592233 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.592049    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ccea4e99fc6cbfe28e390a525c8c585c12cab33ca3a6a176054d9ca4c278799"
Oct 31 01:52:55.811405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.811368    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"]
Oct 31 01:52:55.814229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:55.814205    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2c0de9-openshift-default-5bd7858857-v5nfv"]
Oct 31 01:52:56.845025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:56.844987    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" path="/var/lib/kubelet/pods/c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97/volumes"
Oct 31 01:52:58.373221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373186    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"]
Oct 31 01:52:58.373760 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373743    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c4a02b55-0131-4427-ac05-0c21235dde03" containerName="revoke-keys"
Oct 31 01:52:58.373801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373761    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a02b55-0131-4427-ac05-0c21235dde03" containerName="revoke-keys"
Oct 31 01:52:58.373801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373774    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy"
Oct 31 01:52:58.373801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373779    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy"
Oct 31 01:52:58.373904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373864    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy"
Oct 31 01:52:58.373904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.373877    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a02b55-0131-4427-ac05-0c21235dde03" containerName="revoke-keys"
Oct 31 01:52:58.378516 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.378500    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.433903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.433864    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"]
Oct 31 01:52:58.479699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.479660    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:52:58.479699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.479694    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:52:58.479888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.479720    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:52:58.479888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.479754    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:52:58.517782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.517752    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.517941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.517802    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bz5\" (UniqueName: \"kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.517941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.517883    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.619148 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.619119    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bz5\" (UniqueName: \"kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.619318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.619179    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.619363 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.619332    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.619819 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.619793    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.621723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.621706    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.677795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.677731    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bz5\" (UniqueName: \"kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5\") pod \"maas-api-revoke-sub-a87a32a2-57bwh\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:58.691203 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:58.691181    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:52:59.064915 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:59.064887    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"]
Oct 31 01:52:59.067373 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:52:59.067342    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35c9b2f_e86a_48f0_9a87_79af6bfc6397.slice/crio-6dbd77402da451e62e28a219f3ad6c31988c19aa8ea3b8de2d9ec6de9cbb0a89 WatchSource:0}: Error finding container 6dbd77402da451e62e28a219f3ad6c31988c19aa8ea3b8de2d9ec6de9cbb0a89: Status 404 returned error can't find the container with id 6dbd77402da451e62e28a219f3ad6c31988c19aa8ea3b8de2d9ec6de9cbb0a89
Oct 31 01:52:59.618552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:59.618516    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh" event={"ID":"f35c9b2f-e86a-48f0-9a87-79af6bfc6397","Type":"ContainerStarted","Data":"17004a7385addae8a0983532c3c8022edaf7c6538607bc5acc8623a6f444beb2"}
Oct 31 01:52:59.618947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:59.618571    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh" event={"ID":"f35c9b2f-e86a-48f0-9a87-79af6bfc6397","Type":"ContainerStarted","Data":"6dbd77402da451e62e28a219f3ad6c31988c19aa8ea3b8de2d9ec6de9cbb0a89"}
Oct 31 01:52:59.784871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:52:59.784808    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh" podStartSLOduration=1.784786183 podStartE2EDuration="1.784786183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:59.733098973 +0000 UTC m=+2901.034055551" watchObservedRunningTime="2026-10-31 01:52:59.784786183 +0000 UTC m=+2901.085742762"
Oct 31 01:53:00.629355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:00.629313    5419 generic.go:356] "Generic (PLEG): container finished" podID="f35c9b2f-e86a-48f0-9a87-79af6bfc6397" containerID="17004a7385addae8a0983532c3c8022edaf7c6538607bc5acc8623a6f444beb2" exitCode=0
Oct 31 01:53:00.629748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:00.629376    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh" event={"ID":"f35c9b2f-e86a-48f0-9a87-79af6bfc6397","Type":"ContainerDied","Data":"17004a7385addae8a0983532c3c8022edaf7c6538607bc5acc8623a6f444beb2"}
Oct 31 01:53:01.789016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.788991    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:53:01.856681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.856653    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca\") pod \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") "
Oct 31 01:53:01.856834 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.856790    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5\" (UniqueName: \"kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5\") pod \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") "
Oct 31 01:53:01.856904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.856889    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca\") pod \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\" (UID: \"f35c9b2f-e86a-48f0-9a87-79af6bfc6397\") "
Oct 31 01:53:01.857288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.857264    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca" pod "f35c9b2f-e86a-48f0-9a87-79af6bfc6397" (UID: "f35c9b2f-e86a-48f0-9a87-79af6bfc6397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:01.857566 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.857389    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:01.858920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.858900    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca" pod "f35c9b2f-e86a-48f0-9a87-79af6bfc6397" (UID: "f35c9b2f-e86a-48f0-9a87-79af6bfc6397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:01.859008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.858971    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5" pod "f35c9b2f-e86a-48f0-9a87-79af6bfc6397" (UID: "f35c9b2f-e86a-48f0-9a87-79af6bfc6397"). InnerVolumeSpecName "kube-api-access-h6bz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:01.958546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.958466    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:01.958546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:01.958493    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6bz5\" (UniqueName: \"kubernetes.io/projected/f35c9b2f-e86a-48f0-9a87-79af6bfc6397-kube-api-access-h6bz5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:02.647853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:02.647819    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh" event={"ID":"f35c9b2f-e86a-48f0-9a87-79af6bfc6397","Type":"ContainerDied","Data":"6dbd77402da451e62e28a219f3ad6c31988c19aa8ea3b8de2d9ec6de9cbb0a89"}
Oct 31 01:53:02.647853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:02.647847    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"
Oct 31 01:53:02.648041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:02.647866    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dbd77402da451e62e28a219f3ad6c31988c19aa8ea3b8de2d9ec6de9cbb0a89"
Oct 31 01:53:07.124504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.124473    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"]
Oct 31 01:53:07.124992 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.124978    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f35c9b2f-e86a-48f0-9a87-79af6bfc6397" containerName="revoke-keys"
Oct 31 01:53:07.125039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.124993    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35c9b2f-e86a-48f0-9a87-79af6bfc6397" containerName="revoke-keys"
Oct 31 01:53:07.125039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.125004    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy"
Oct 31 01:53:07.125039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.125009    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy"
Oct 31 01:53:07.125159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.125096    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fdbaaf-1f43-4baf-a00d-4fe6d9e1fb97" containerName="istio-proxy"
Oct 31 01:53:07.125159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.125110    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35c9b2f-e86a-48f0-9a87-79af6bfc6397" containerName="revoke-keys"
Oct 31 01:53:07.128484 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.128469    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.186242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.186211    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"]
Oct 31 01:53:07.219041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.219010    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.219250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.219068    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftswm\" (UniqueName: \"kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.219250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.219213    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.231224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.231189    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:07.231224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.231207    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:07.231224 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.231207    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:07.231426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.231237    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:07.320689 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.320653    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftswm\" (UniqueName: \"kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.320902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.320706    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.320902 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.320807    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.321411 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.321390    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.323344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.323326    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.377631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.377558    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftswm\" (UniqueName: \"kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm\") pod \"maas-api-revoke-sub-68a36982-gsw6c\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.441297 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.441263    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:07.813257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:07.813229    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"]
Oct 31 01:53:07.814882 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:07.814854    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d4525f_8369_400b_bf1b_48c0d7b616f3.slice/crio-91f38c483c6ad61a587051c75e546a47b7ade5c694f167c4817502b05dc239e8 WatchSource:0}: Error finding container 91f38c483c6ad61a587051c75e546a47b7ade5c694f167c4817502b05dc239e8: Status 404 returned error can't find the container with id 91f38c483c6ad61a587051c75e546a47b7ade5c694f167c4817502b05dc239e8
Oct 31 01:53:08.008801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.008770    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"]
Oct 31 01:53:08.014581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.014559    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.116879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.116847    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a8e157-openshift-default-dockercfg-f24hn\""
Oct 31 01:53:08.119587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.119562    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"]
Oct 31 01:53:08.129326 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129305    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129375    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129435    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129507    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129537    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2ffs\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129616    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129659    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129768    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.129832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.129803    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231038    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231072 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231074    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2ffs\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231121    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231164    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231304    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231381    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231435    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231513    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231516    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231560    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231642    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231692    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231869    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.231946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.231877    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.233845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.233815    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.234036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.234020    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.288291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.288266    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.288428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.288381    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2ffs\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.328395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.328354    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:08.688039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.688008    5419 generic.go:356] "Generic (PLEG): container finished" podID="16d4525f-8369-400b-bf1b-48c0d7b616f3" containerID="b5744c5390d2ee57c3a404fa0c61dbdaf66533c1aca9428bccfc7efbd6959d1e" exitCode=0
Oct 31 01:53:08.688212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.688074    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c" event={"ID":"16d4525f-8369-400b-bf1b-48c0d7b616f3","Type":"ContainerDied","Data":"b5744c5390d2ee57c3a404fa0c61dbdaf66533c1aca9428bccfc7efbd6959d1e"}
Oct 31 01:53:08.688212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.688111    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c" event={"ID":"16d4525f-8369-400b-bf1b-48c0d7b616f3","Type":"ContainerStarted","Data":"91f38c483c6ad61a587051c75e546a47b7ade5c694f167c4817502b05dc239e8"}
Oct 31 01:53:08.755765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.755735    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"]
Oct 31 01:53:08.756876 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:08.756852    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c9953f_0348_4ca0_a0e5_b0d4e5226580.slice/crio-b68aca2ec0676b9bbeb0a1fdcc1aab8ef5962cf2fb7383b0ec74544fab7dc7e4 WatchSource:0}: Error finding container b68aca2ec0676b9bbeb0a1fdcc1aab8ef5962cf2fb7383b0ec74544fab7dc7e4: Status 404 returned error can't find the container with id b68aca2ec0676b9bbeb0a1fdcc1aab8ef5962cf2fb7383b0ec74544fab7dc7e4
Oct 31 01:53:08.759373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.759334    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:53:08.759475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.759400    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:53:08.759475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:08.759429    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:53:09.705938 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:09.705904    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" event={"ID":"c2c9953f-0348-4ca0-a0e5-b0d4e5226580","Type":"ContainerStarted","Data":"169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901"}
Oct 31 01:53:09.706395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:09.705965    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" event={"ID":"c2c9953f-0348-4ca0-a0e5-b0d4e5226580","Type":"ContainerStarted","Data":"b68aca2ec0676b9bbeb0a1fdcc1aab8ef5962cf2fb7383b0ec74544fab7dc7e4"}
Oct 31 01:53:09.823931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:09.823870    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podStartSLOduration=2.823854023 podStartE2EDuration="2.823854023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:09.821501301 +0000 UTC m=+2911.122457881" watchObservedRunningTime="2026-10-31 01:53:09.823854023 +0000 UTC m=+2911.124810600"
Oct 31 01:53:09.863692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:09.863661    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:10.057211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.057178    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca\") pod \"16d4525f-8369-400b-bf1b-48c0d7b616f3\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") "
Oct 31 01:53:10.057426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.057221    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm\" (UniqueName: \"kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm\") pod \"16d4525f-8369-400b-bf1b-48c0d7b616f3\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") "
Oct 31 01:53:10.057426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.057339    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca\") pod \"16d4525f-8369-400b-bf1b-48c0d7b616f3\" (UID: \"16d4525f-8369-400b-bf1b-48c0d7b616f3\") "
Oct 31 01:53:10.057717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.057695    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca" pod "16d4525f-8369-400b-bf1b-48c0d7b616f3" (UID: "16d4525f-8369-400b-bf1b-48c0d7b616f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:10.059655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.059634    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca" pod "16d4525f-8369-400b-bf1b-48c0d7b616f3" (UID: "16d4525f-8369-400b-bf1b-48c0d7b616f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:10.059728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.059657    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm" pod "16d4525f-8369-400b-bf1b-48c0d7b616f3" (UID: "16d4525f-8369-400b-bf1b-48c0d7b616f3"). InnerVolumeSpecName "kube-api-access-ftswm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:10.159200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.159159    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:10.159200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.159195    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16d4525f-8369-400b-bf1b-48c0d7b616f3-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:10.159421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.159208    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftswm\" (UniqueName: \"kubernetes.io/projected/16d4525f-8369-400b-bf1b-48c0d7b616f3-kube-api-access-ftswm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:10.329385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.329299    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:10.330786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.330759    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 31 01:53:10.330926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.330815    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 31 01:53:10.714164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.714071    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"
Oct 31 01:53:10.714164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.714105    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c" event={"ID":"16d4525f-8369-400b-bf1b-48c0d7b616f3","Type":"ContainerDied","Data":"91f38c483c6ad61a587051c75e546a47b7ade5c694f167c4817502b05dc239e8"}
Oct 31 01:53:10.714629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:10.714169    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91f38c483c6ad61a587051c75e546a47b7ade5c694f167c4817502b05dc239e8"
Oct 31 01:53:11.329310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:11.329272    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 31 01:53:11.329528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:11.329331    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 31 01:53:12.329271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:12.329239    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 31 01:53:12.329690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:12.329300    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 31 01:53:12.534433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:12.534399    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:53:13.329789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:13.329753    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 31 01:53:13.330236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:13.329811    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 31 01:53:14.111962 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.111929    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"]
Oct 31 01:53:14.112565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.112548    5419 cpu_manager.go:414] "containerMap: removing container" podUID="16d4525f-8369-400b-bf1b-48c0d7b616f3" containerName="revoke-keys"
Oct 31 01:53:14.112643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.112566    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d4525f-8369-400b-bf1b-48c0d7b616f3" containerName="revoke-keys"
Oct 31 01:53:14.112681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.112667    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d4525f-8369-400b-bf1b-48c0d7b616f3" containerName="revoke-keys"
Oct 31 01:53:14.116243 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.116224    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.127822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.127798    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.127929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.127835    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp58x\" (UniqueName: \"kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.127929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.127907    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.170148 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.170116    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:14.218174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.218095    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:14.218174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.218094    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:14.218174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.218125    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:14.221374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.221349    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"]
Oct 31 01:53:14.228975 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.228949    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.229063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.229005    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp58x\" (UniqueName: \"kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.229114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.229061    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.229674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.229650    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.231735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.231709    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.301191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.301156    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp58x\" (UniqueName: \"kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x\") pod \"maas-api-revoke-sub-3d13bcd5-r24gv\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.328926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.328900    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 31 01:53:14.329088 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.328951    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 31 01:53:14.429252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.429168    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:14.619681 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:53:14.619612    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:53:14.836074 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:14.835413    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d07ee7_e157_409a_9773_1ba958dd967e.slice/crio-f34bf86d1aa000d99cd75aab42bcf0edfc8156275069b3443a502fcaa39f544b WatchSource:0}: Error finding container f34bf86d1aa000d99cd75aab42bcf0edfc8156275069b3443a502fcaa39f544b: Status 404 returned error can't find the container with id f34bf86d1aa000d99cd75aab42bcf0edfc8156275069b3443a502fcaa39f544b
Oct 31 01:53:14.846625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:14.846600    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"]
Oct 31 01:53:15.328910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:15.328869    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 31 01:53:15.329125 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:15.328937    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 31 01:53:15.753052 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:15.753006    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv" event={"ID":"61d07ee7-e157-409a-9773-1ba958dd967e","Type":"ContainerStarted","Data":"48a282bd93930f4eab72517e1f9732e69184ac9f1c79273a42ea23a018e306c5"}
Oct 31 01:53:15.753585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:15.753078    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv" event={"ID":"61d07ee7-e157-409a-9773-1ba958dd967e","Type":"ContainerStarted","Data":"f34bf86d1aa000d99cd75aab42bcf0edfc8156275069b3443a502fcaa39f544b"}
Oct 31 01:53:15.868541 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:15.868481    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv" podStartSLOduration=1.868466346 podStartE2EDuration="1.868466346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:15.865960813 +0000 UTC m=+2917.166917418" watchObservedRunningTime="2026-10-31 01:53:15.868466346 +0000 UTC m=+2917.169422922"
Oct 31 01:53:16.763338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:16.763304    5419 generic.go:356] "Generic (PLEG): container finished" podID="61d07ee7-e157-409a-9773-1ba958dd967e" containerID="48a282bd93930f4eab72517e1f9732e69184ac9f1c79273a42ea23a018e306c5" exitCode=0
Oct 31 01:53:16.763800 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:16.763382    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv" event={"ID":"61d07ee7-e157-409a-9773-1ba958dd967e","Type":"ContainerDied","Data":"48a282bd93930f4eab72517e1f9732e69184ac9f1c79273a42ea23a018e306c5"}
Oct 31 01:53:17.370309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.370264    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:53:17.370507 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.370325    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:53:17.520655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.520625    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:17.521117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.521096    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:17.521917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.521898    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 01:53:17.942477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.942449    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:17.977167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.977121    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca\") pod \"61d07ee7-e157-409a-9773-1ba958dd967e\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") "
Oct 31 01:53:17.977302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.977192    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca\") pod \"61d07ee7-e157-409a-9773-1ba958dd967e\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") "
Oct 31 01:53:17.977302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.977289    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x\" (UniqueName: \"kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x\") pod \"61d07ee7-e157-409a-9773-1ba958dd967e\" (UID: \"61d07ee7-e157-409a-9773-1ba958dd967e\") "
Oct 31 01:53:17.977530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.977505    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca" pod "61d07ee7-e157-409a-9773-1ba958dd967e" (UID: "61d07ee7-e157-409a-9773-1ba958dd967e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:17.977860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.977808    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:17.979393 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.979371    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca" pod "61d07ee7-e157-409a-9773-1ba958dd967e" (UID: "61d07ee7-e157-409a-9773-1ba958dd967e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:17.979517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:17.979502    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x" pod "61d07ee7-e157-409a-9773-1ba958dd967e" (UID: "61d07ee7-e157-409a-9773-1ba958dd967e"). InnerVolumeSpecName "kube-api-access-gp58x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:18.078896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.078820    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp58x\" (UniqueName: \"kubernetes.io/projected/61d07ee7-e157-409a-9773-1ba958dd967e-kube-api-access-gp58x\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:18.078896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.078847    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d07ee7-e157-409a-9773-1ba958dd967e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:18.776788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.776760    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"
Oct 31 01:53:18.776976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.776787    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv" event={"ID":"61d07ee7-e157-409a-9773-1ba958dd967e","Type":"ContainerDied","Data":"f34bf86d1aa000d99cd75aab42bcf0edfc8156275069b3443a502fcaa39f544b"}
Oct 31 01:53:18.776976 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.776862    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f34bf86d1aa000d99cd75aab42bcf0edfc8156275069b3443a502fcaa39f544b"
Oct 31 01:53:18.823847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.823819    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"]
Oct 31 01:53:18.824356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.824342    5419 cpu_manager.go:414] "containerMap: removing container" podUID="61d07ee7-e157-409a-9773-1ba958dd967e" containerName="revoke-keys"
Oct 31 01:53:18.824400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.824358    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d07ee7-e157-409a-9773-1ba958dd967e" containerName="revoke-keys"
Oct 31 01:53:18.824464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.824455    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d07ee7-e157-409a-9773-1ba958dd967e" containerName="revoke-keys"
Oct 31 01:53:18.827913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.827896    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.880042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.880012    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:18.880268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.880165    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:18.880355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.880278    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:18.880355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.880334    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:18.883754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.883731    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"]
Oct 31 01:53:18.886173 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.886132    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.886654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.886613    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdw2p\" (UniqueName: \"kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.887235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.887211    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.988581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.988547    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.988990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.988604    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdw2p\" (UniqueName: \"kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.988990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.988663    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.989331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.989312    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:18.991092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:18.991057    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:19.045288 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:19.045223    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdw2p\" (UniqueName: \"kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p\") pod \"maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:19.143572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:19.143540    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:19.516916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:19.516891    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"]
Oct 31 01:53:19.518111 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:19.518082    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1992f5_1016_4a04_a9eb_f523ca855ab4.slice/crio-f44c802fc6c2a97812a07b0a54a8b295b9948a4f9cf66df2f0a8b6b04c4ce2e2 WatchSource:0}: Error finding container f44c802fc6c2a97812a07b0a54a8b295b9948a4f9cf66df2f0a8b6b04c4ce2e2: Status 404 returned error can't find the container with id f44c802fc6c2a97812a07b0a54a8b295b9948a4f9cf66df2f0a8b6b04c4ce2e2
Oct 31 01:53:19.785025 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:19.785000    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9" event={"ID":"9c1992f5-1016-4a04-a9eb-f523ca855ab4","Type":"ContainerStarted","Data":"f44c802fc6c2a97812a07b0a54a8b295b9948a4f9cf66df2f0a8b6b04c4ce2e2"}
Oct 31 01:53:20.793718 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:20.793681    5419 generic.go:356] "Generic (PLEG): container finished" podID="9c1992f5-1016-4a04-a9eb-f523ca855ab4" containerID="593c93c16ea1df1186b1ba68164eec363155ff15528530402707f7db2e2d4bb7" exitCode=0
Oct 31 01:53:20.794096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:20.793768    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9" event={"ID":"9c1992f5-1016-4a04-a9eb-f523ca855ab4","Type":"ContainerDied","Data":"593c93c16ea1df1186b1ba68164eec363155ff15528530402707f7db2e2d4bb7"}
Oct 31 01:53:21.948311 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:21.948286    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:22.026398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.026365    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p\" (UniqueName: \"kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p\") pod \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") "
Oct 31 01:53:22.026542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.026443    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca\") pod \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") "
Oct 31 01:53:22.026542 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.026475    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca\") pod \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\" (UID: \"9c1992f5-1016-4a04-a9eb-f523ca855ab4\") "
Oct 31 01:53:22.026878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.026852    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca" pod "9c1992f5-1016-4a04-a9eb-f523ca855ab4" (UID: "9c1992f5-1016-4a04-a9eb-f523ca855ab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:22.028657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.028635    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca" pod "9c1992f5-1016-4a04-a9eb-f523ca855ab4" (UID: "9c1992f5-1016-4a04-a9eb-f523ca855ab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:22.028769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.028659    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p" pod "9c1992f5-1016-4a04-a9eb-f523ca855ab4" (UID: "9c1992f5-1016-4a04-a9eb-f523ca855ab4"). InnerVolumeSpecName "kube-api-access-bdw2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:22.127502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.127426    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdw2p\" (UniqueName: \"kubernetes.io/projected/9c1992f5-1016-4a04-a9eb-f523ca855ab4-kube-api-access-bdw2p\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:22.127502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.127452    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:22.127502 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.127463    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1992f5-1016-4a04-a9eb-f523ca855ab4-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:22.810704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.810668    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"
Oct 31 01:53:22.810704 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.810681    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9" event={"ID":"9c1992f5-1016-4a04-a9eb-f523ca855ab4","Type":"ContainerDied","Data":"f44c802fc6c2a97812a07b0a54a8b295b9948a4f9cf66df2f0a8b6b04c4ce2e2"}
Oct 31 01:53:22.810967 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:22.810756    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f44c802fc6c2a97812a07b0a54a8b295b9948a4f9cf66df2f0a8b6b04c4ce2e2"
Oct 31 01:53:23.572282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.572241    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"]
Oct 31 01:53:23.572778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.572734    5419 cpu_manager.go:414] "containerMap: removing container" podUID="9c1992f5-1016-4a04-a9eb-f523ca855ab4" containerName="revoke-keys"
Oct 31 01:53:23.572778 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.572746    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1992f5-1016-4a04-a9eb-f523ca855ab4" containerName="revoke-keys"
Oct 31 01:53:23.572899 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.572870    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1992f5-1016-4a04-a9eb-f523ca855ab4" containerName="revoke-keys"
Oct 31 01:53:23.576328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.576308    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.632571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.632532    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"]
Oct 31 01:53:23.644365 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.644336    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smpz\" (UniqueName: \"kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.644523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.644371    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.644523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.644462    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.678184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.678129    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:23.678184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.678160    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:23.678379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.678199    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:23.678379 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.678291    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:23.745666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.745633    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smpz\" (UniqueName: \"kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.745666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.745665    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.745912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.745695    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.746312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.746292    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.748149 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.748120    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.801995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.801967    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smpz\" (UniqueName: \"kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz\") pod \"maas-api-revoke-sub-13843630-lp2nt\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:23.891854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:23.891774    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:24.264057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:24.264029    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"]
Oct 31 01:53:24.265736 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:24.265711    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178cc66b_3582_4bde_b580_948c67bb8a96.slice/crio-71b6c11091860807c34e340a3f9bcf73964df0bf11710a92ad838769b65756ea WatchSource:0}: Error finding container 71b6c11091860807c34e340a3f9bcf73964df0bf11710a92ad838769b65756ea: Status 404 returned error can't find the container with id 71b6c11091860807c34e340a3f9bcf73964df0bf11710a92ad838769b65756ea
Oct 31 01:53:24.825394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:24.825354    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt" event={"ID":"178cc66b-3582-4bde-b580-948c67bb8a96","Type":"ContainerStarted","Data":"c5ce03a8763e062ef9cb4c07d0f07f775f6558f5682258ec339f47e2c32ab6c4"}
Oct 31 01:53:24.825785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:24.825412    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt" event={"ID":"178cc66b-3582-4bde-b580-948c67bb8a96","Type":"ContainerStarted","Data":"71b6c11091860807c34e340a3f9bcf73964df0bf11710a92ad838769b65756ea"}
Oct 31 01:53:24.939791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:24.939694    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt" podStartSLOduration=1.939679027 podStartE2EDuration="1.939679027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:24.938921489 +0000 UTC m=+2926.239878060" watchObservedRunningTime="2026-10-31 01:53:24.939679027 +0000 UTC m=+2926.240635603"
Oct 31 01:53:25.840622 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:25.840586    5419 generic.go:356] "Generic (PLEG): container finished" podID="178cc66b-3582-4bde-b580-948c67bb8a96" containerID="c5ce03a8763e062ef9cb4c07d0f07f775f6558f5682258ec339f47e2c32ab6c4" exitCode=0
Oct 31 01:53:25.841015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:25.840674    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt" event={"ID":"178cc66b-3582-4bde-b580-948c67bb8a96","Type":"ContainerDied","Data":"c5ce03a8763e062ef9cb4c07d0f07f775f6558f5682258ec339f47e2c32ab6c4"}
Oct 31 01:53:26.463965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.463930    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"]
Oct 31 01:53:26.468052 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.468036    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.573599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.573560    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"]
Oct 31 01:53:26.577868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.577837    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.577904    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.577988    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.578072    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.578151    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.578326    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.578391    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.578473    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.578561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.578518    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtx4\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.679947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.679914    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.679947 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.679951    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.679982    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680013    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680087    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680166    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680239    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680266    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtx4\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680295    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680803 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680749    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680757    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680870    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.680924 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680912    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.681070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.680976    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.682910 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.682878    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.682963 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.682888    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.736811 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.736777    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.736944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.736812    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtx4\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.781325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.781297    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:26.996972 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:26.996916    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:27.086846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.086819    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca\") pod \"178cc66b-3582-4bde-b580-948c67bb8a96\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") "
Oct 31 01:53:27.087012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.086866    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca\") pod \"178cc66b-3582-4bde-b580-948c67bb8a96\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") "
Oct 31 01:53:27.087012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.086945    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz\" (UniqueName: \"kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz\") pod \"178cc66b-3582-4bde-b580-948c67bb8a96\" (UID: \"178cc66b-3582-4bde-b580-948c67bb8a96\") "
Oct 31 01:53:27.087330 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.087302    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca" pod "178cc66b-3582-4bde-b580-948c67bb8a96" (UID: "178cc66b-3582-4bde-b580-948c67bb8a96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:27.087453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.087402    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:27.089005 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.088981    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca" pod "178cc66b-3582-4bde-b580-948c67bb8a96" (UID: "178cc66b-3582-4bde-b580-948c67bb8a96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:27.089108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.089078    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz" pod "178cc66b-3582-4bde-b580-948c67bb8a96" (UID: "178cc66b-3582-4bde-b580-948c67bb8a96"). InnerVolumeSpecName "kube-api-access-4smpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:27.188757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.188725    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smpz\" (UniqueName: \"kubernetes.io/projected/178cc66b-3582-4bde-b580-948c67bb8a96-kube-api-access-4smpz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:27.188757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.188754    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178cc66b-3582-4bde-b580-948c67bb8a96-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:27.210023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.209997    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"]
Oct 31 01:53:27.210644 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:27.210620    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a8db1b_fdcc_469d_88bc_1172e8860382.slice/crio-7f62ea6a7ec73565e5f8d18a3278dd9f5ff7985e9a6b9835e314834888bd721a WatchSource:0}: Error finding container 7f62ea6a7ec73565e5f8d18a3278dd9f5ff7985e9a6b9835e314834888bd721a: Status 404 returned error can't find the container with id 7f62ea6a7ec73565e5f8d18a3278dd9f5ff7985e9a6b9835e314834888bd721a
Oct 31 01:53:27.213114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.213085    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:53:27.213202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.213167    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:53:27.213244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.213199    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:53:27.867706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.867672    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt" event={"ID":"178cc66b-3582-4bde-b580-948c67bb8a96","Type":"ContainerDied","Data":"71b6c11091860807c34e340a3f9bcf73964df0bf11710a92ad838769b65756ea"}
Oct 31 01:53:27.867883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.867718    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71b6c11091860807c34e340a3f9bcf73964df0bf11710a92ad838769b65756ea"
Oct 31 01:53:27.867883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.867679    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"
Oct 31 01:53:27.869517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.869483    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" event={"ID":"f2a8db1b-fdcc-469d-88bc-1172e8860382","Type":"ContainerStarted","Data":"6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25"}
Oct 31 01:53:27.869653 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:27.869536    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" event={"ID":"f2a8db1b-fdcc-469d-88bc-1172e8860382","Type":"ContainerStarted","Data":"7f62ea6a7ec73565e5f8d18a3278dd9f5ff7985e9a6b9835e314834888bd721a"}
Oct 31 01:53:28.006922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.006861    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podStartSLOduration=2.006843011 podStartE2EDuration="2.006843011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:27.992654833 +0000 UTC m=+2929.293611415" watchObservedRunningTime="2026-10-31 01:53:28.006843011 +0000 UTC m=+2929.307799592"
Oct 31 01:53:28.590163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.585022    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"]
Oct 31 01:53:28.590163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.585845    5419 cpu_manager.go:414] "containerMap: removing container" podUID="178cc66b-3582-4bde-b580-948c67bb8a96" containerName="revoke-keys"
Oct 31 01:53:28.590163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.585866    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178cc66b-3582-4bde-b580-948c67bb8a96" containerName="revoke-keys"
Oct 31 01:53:28.590163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.586062    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="178cc66b-3582-4bde-b580-948c67bb8a96" containerName="revoke-keys"
Oct 31 01:53:28.605310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.605270    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.640738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.640696    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"]
Oct 31 01:53:28.657688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.657648    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:28.657941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.657733    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:28.657941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.657860    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:28.658076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.657967    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:28.709445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.709390    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.709625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.709549    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57b6\" (UniqueName: \"kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.709694 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.709638    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.791804 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.791774    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:28.792211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.792189    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:53:28.792387 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.792362    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:53:28.811086 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.811062    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57b6\" (UniqueName: \"kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.811258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.811123    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.811258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.811238    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.811795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.811772    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.813823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.813802    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.867626 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.867551    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57b6\" (UniqueName: \"kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6\") pod \"maas-api-revoke-sub-523340b7-dmcw2\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:28.919453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:28.919423    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:29.308999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:29.308970    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"]
Oct 31 01:53:29.310461 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:29.310434    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0763834b_10a9_43f5_b308_1b1854a01ed9.slice/crio-7968002453e5a5cdd36a01c7f5107e02eb7b53a5893339c3b0e37c726cdb354a WatchSource:0}: Error finding container 7968002453e5a5cdd36a01c7f5107e02eb7b53a5893339c3b0e37c726cdb354a: Status 404 returned error can't find the container with id 7968002453e5a5cdd36a01c7f5107e02eb7b53a5893339c3b0e37c726cdb354a
Oct 31 01:53:29.799465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:29.799423    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:53:29.799697 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:29.799491    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:53:29.911178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:29.911120    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2" event={"ID":"0763834b-10a9-43f5-b308-1b1854a01ed9","Type":"ContainerStarted","Data":"52b5afd75bcf5cf71a3e4c7a11162789a48e69e639bdd6a24a4aba3d65e3a126"}
Oct 31 01:53:29.911453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:29.911428    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2" event={"ID":"0763834b-10a9-43f5-b308-1b1854a01ed9","Type":"ContainerStarted","Data":"7968002453e5a5cdd36a01c7f5107e02eb7b53a5893339c3b0e37c726cdb354a"}
Oct 31 01:53:30.026036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:30.025981    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2" podStartSLOduration=2.025966173 podStartE2EDuration="2.025966173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:30.024622509 +0000 UTC m=+2931.325579086" watchObservedRunningTime="2026-10-31 01:53:30.025966173 +0000 UTC m=+2931.326922807"
Oct 31 01:53:30.803553 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:30.803522    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:53:30.804198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:30.804116    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:53:30.956757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:30.956721    5419 generic.go:356] "Generic (PLEG): container finished" podID="0763834b-10a9-43f5-b308-1b1854a01ed9" containerID="52b5afd75bcf5cf71a3e4c7a11162789a48e69e639bdd6a24a4aba3d65e3a126" exitCode=0
Oct 31 01:53:30.956950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:30.956933    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2" event={"ID":"0763834b-10a9-43f5-b308-1b1854a01ed9","Type":"ContainerDied","Data":"52b5afd75bcf5cf71a3e4c7a11162789a48e69e639bdd6a24a4aba3d65e3a126"}
Oct 31 01:53:31.799306 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:31.799261    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:53:31.799561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:31.799341    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:53:32.146253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.146226    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:32.255537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.255504    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca\") pod \"0763834b-10a9-43f5-b308-1b1854a01ed9\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") "
Oct 31 01:53:32.255708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.255646    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca\") pod \"0763834b-10a9-43f5-b308-1b1854a01ed9\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") "
Oct 31 01:53:32.255708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.255681    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6\" (UniqueName: \"kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6\") pod \"0763834b-10a9-43f5-b308-1b1854a01ed9\" (UID: \"0763834b-10a9-43f5-b308-1b1854a01ed9\") "
Oct 31 01:53:32.256011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.255969    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca" pod "0763834b-10a9-43f5-b308-1b1854a01ed9" (UID: "0763834b-10a9-43f5-b308-1b1854a01ed9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:32.256263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.256245    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:32.257919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.257891    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca" pod "0763834b-10a9-43f5-b308-1b1854a01ed9" (UID: "0763834b-10a9-43f5-b308-1b1854a01ed9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:32.257919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.257897    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6" pod "0763834b-10a9-43f5-b308-1b1854a01ed9" (UID: "0763834b-10a9-43f5-b308-1b1854a01ed9"). InnerVolumeSpecName "kube-api-access-j57b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:32.357262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.357212    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j57b6\" (UniqueName: \"kubernetes.io/projected/0763834b-10a9-43f5-b308-1b1854a01ed9-kube-api-access-j57b6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:32.357262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.357258    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0763834b-10a9-43f5-b308-1b1854a01ed9-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:32.772323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.772285    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"]
Oct 31 01:53:32.798445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.797439    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:53:32.798445 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.797515    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:53:32.970770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.970736    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2" event={"ID":"0763834b-10a9-43f5-b308-1b1854a01ed9","Type":"ContainerDied","Data":"7968002453e5a5cdd36a01c7f5107e02eb7b53a5893339c3b0e37c726cdb354a"}
Oct 31 01:53:32.970770 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.970770    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"
Oct 31 01:53:32.970988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:32.970795    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7968002453e5a5cdd36a01c7f5107e02eb7b53a5893339c3b0e37c726cdb354a"
Oct 31 01:53:33.782331 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:33.782299    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 01:53:33.782739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:33.782349    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 01:53:34.825496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:34.825462    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:34.825987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:34.825854    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:34.826679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:34.826657    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 01:53:37.773250 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:37.773185    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerName="payload-processing" containerID="cri-o://61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528" gracePeriod=25
Oct 31 01:53:39.579871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.579837    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"]
Oct 31 01:53:39.580705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.580686    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0763834b-10a9-43f5-b308-1b1854a01ed9" containerName="revoke-keys"
Oct 31 01:53:39.580756 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.580709    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0763834b-10a9-43f5-b308-1b1854a01ed9" containerName="revoke-keys"
Oct 31 01:53:39.580941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.580927    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0763834b-10a9-43f5-b308-1b1854a01ed9" containerName="revoke-keys"
Oct 31 01:53:39.585077 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.585052    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.639674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.639646    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"]
Oct 31 01:53:39.665736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.665706    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmh6\" (UniqueName: \"kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.665873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.665750    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.665973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.665953    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.685302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.685282    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:39.685408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.685354    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:39.685408 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.685394    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:39.686988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.686975    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:39.766765 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.766734    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmh6\" (UniqueName: \"kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.766965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.766782    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.766965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.766923    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.767491 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.767471    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.769380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.769357    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.824575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.824539    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmh6\" (UniqueName: \"kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6\") pod \"maas-api-revoke-sub-5bac52fa-h92b5\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:39.898901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:39.898825    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:40.280883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:40.280850    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"]
Oct 31 01:53:40.281801 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:40.281774    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60f2b84_b55e_4b1a_a826_4a630ef10a55.slice/crio-60753a5f4e0326fa0912e4c6191ba5b98492489c9b8cf2456cd2631d8c7d99f1 WatchSource:0}: Error finding container 60753a5f4e0326fa0912e4c6191ba5b98492489c9b8cf2456cd2631d8c7d99f1: Status 404 returned error can't find the container with id 60753a5f4e0326fa0912e4c6191ba5b98492489c9b8cf2456cd2631d8c7d99f1
Oct 31 01:53:40.944563 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:40.944483    5419 scope.go:122] "RemoveContainer" containerID="fb7122dddfee1494ce7ea250912784c853b1081041f1951d9e167146f57d55c2"
Oct 31 01:53:41.033893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:41.033848    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5" event={"ID":"e60f2b84-b55e-4b1a-a826-4a630ef10a55","Type":"ContainerStarted","Data":"36c5c0a012c1bb4b2878073070f2a5eea9a3e4a1340072a21f0bdf0c068acae2"}
Oct 31 01:53:41.034063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:41.033909    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5" event={"ID":"e60f2b84-b55e-4b1a-a826-4a630ef10a55","Type":"ContainerStarted","Data":"60753a5f4e0326fa0912e4c6191ba5b98492489c9b8cf2456cd2631d8c7d99f1"}
Oct 31 01:53:41.151314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:41.151266    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5" podStartSLOduration=2.151253887 podStartE2EDuration="2.151253887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:41.147935475 +0000 UTC m=+2942.448892052" watchObservedRunningTime="2026-10-31 01:53:41.151253887 +0000 UTC m=+2942.452210463"
Oct 31 01:53:42.042613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:42.042570    5419 generic.go:356] "Generic (PLEG): container finished" podID="e60f2b84-b55e-4b1a-a826-4a630ef10a55" containerID="36c5c0a012c1bb4b2878073070f2a5eea9a3e4a1340072a21f0bdf0c068acae2" exitCode=0
Oct 31 01:53:42.043050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:42.042658    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5" event={"ID":"e60f2b84-b55e-4b1a-a826-4a630ef10a55","Type":"ContainerDied","Data":"36c5c0a012c1bb4b2878073070f2a5eea9a3e4a1340072a21f0bdf0c068acae2"}
Oct 31 01:53:43.184744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.184720    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:43.311323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.311245    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6\" (UniqueName: \"kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6\") pod \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") "
Oct 31 01:53:43.311503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.311311    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca\") pod \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") "
Oct 31 01:53:43.311679 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.311659    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca\") pod \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\" (UID: \"e60f2b84-b55e-4b1a-a826-4a630ef10a55\") "
Oct 31 01:53:43.312017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.311985    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca" pod "e60f2b84-b55e-4b1a-a826-4a630ef10a55" (UID: "e60f2b84-b55e-4b1a-a826-4a630ef10a55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:43.312206 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.312191    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.314050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.314019    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca" pod "e60f2b84-b55e-4b1a-a826-4a630ef10a55" (UID: "e60f2b84-b55e-4b1a-a826-4a630ef10a55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:43.314050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.314046    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6" pod "e60f2b84-b55e-4b1a-a826-4a630ef10a55" (UID: "e60f2b84-b55e-4b1a-a826-4a630ef10a55"). InnerVolumeSpecName "kube-api-access-mwmh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:43.336988 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.336961    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"]
Oct 31 01:53:43.337835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.337813    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e60f2b84-b55e-4b1a-a826-4a630ef10a55" containerName="revoke-keys"
Oct 31 01:53:43.337928 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.337838    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e60f2b84-b55e-4b1a-a826-4a630ef10a55" containerName="revoke-keys"
Oct 31 01:53:43.338020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.338004    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e60f2b84-b55e-4b1a-a826-4a630ef10a55" containerName="revoke-keys"
Oct 31 01:53:43.342064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.342039    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.409690 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.409647    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"]
Oct 31 01:53:43.413831 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.413796    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.413974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.413875    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.414154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.414111    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86xz\" (UniqueName: \"kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.414293 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.414275    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmh6\" (UniqueName: \"kubernetes.io/projected/e60f2b84-b55e-4b1a-a826-4a630ef10a55-kube-api-access-mwmh6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.414352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.414298    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e60f2b84-b55e-4b1a-a826-4a630ef10a55-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:43.515846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.515810    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.516019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.515911    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86xz\" (UniqueName: \"kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.516019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.515980    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.516597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.516570    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.518364 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.518341    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.572703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.572633    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86xz\" (UniqueName: \"kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz\") pod \"maas-api-revoke-sub-7f7313c2-prfrh\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:43.656912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:43.656884    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:44.031517 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:44.031487    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"]
Oct 31 01:53:44.032756 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:44.032733    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17153e24_fc3f_4582_b071_68db4cfdf0cf.slice/crio-7020cde0ab9e4a6a16033da94d73f495ed0e69c460b78827ecc72f98e59eee26 WatchSource:0}: Error finding container 7020cde0ab9e4a6a16033da94d73f495ed0e69c460b78827ecc72f98e59eee26: Status 404 returned error can't find the container with id 7020cde0ab9e4a6a16033da94d73f495ed0e69c460b78827ecc72f98e59eee26
Oct 31 01:53:44.057604 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:44.057579    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"
Oct 31 01:53:44.057720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:44.057618    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5" event={"ID":"e60f2b84-b55e-4b1a-a826-4a630ef10a55","Type":"ContainerDied","Data":"60753a5f4e0326fa0912e4c6191ba5b98492489c9b8cf2456cd2631d8c7d99f1"}
Oct 31 01:53:44.057720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:44.057673    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60753a5f4e0326fa0912e4c6191ba5b98492489c9b8cf2456cd2631d8c7d99f1"
Oct 31 01:53:44.058909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:44.058886    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh" event={"ID":"17153e24-fc3f-4582-b071-68db4cfdf0cf","Type":"ContainerStarted","Data":"7020cde0ab9e4a6a16033da94d73f495ed0e69c460b78827ecc72f98e59eee26"}
Oct 31 01:53:44.584391 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:53:44.584304    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:53:44.852430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:44.852354    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:45.067245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:45.067210    5419 generic.go:356] "Generic (PLEG): container finished" podID="17153e24-fc3f-4582-b071-68db4cfdf0cf" containerID="7a4533a4cbe7c7438642c1f49d10369325fa4146684dd63663422696914ef62a" exitCode=0
Oct 31 01:53:45.067457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:45.067289    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh" event={"ID":"17153e24-fc3f-4582-b071-68db4cfdf0cf","Type":"ContainerDied","Data":"7a4533a4cbe7c7438642c1f49d10369325fa4146684dd63663422696914ef62a"}
Oct 31 01:53:46.230548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.230516    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:46.357054 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.356968    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca\") pod \"17153e24-fc3f-4582-b071-68db4cfdf0cf\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") "
Oct 31 01:53:46.357251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.357073    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz\" (UniqueName: \"kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz\") pod \"17153e24-fc3f-4582-b071-68db4cfdf0cf\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") "
Oct 31 01:53:46.357338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.357240    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca\") pod \"17153e24-fc3f-4582-b071-68db4cfdf0cf\" (UID: \"17153e24-fc3f-4582-b071-68db4cfdf0cf\") "
Oct 31 01:53:46.357540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.357510    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca" pod "17153e24-fc3f-4582-b071-68db4cfdf0cf" (UID: "17153e24-fc3f-4582-b071-68db4cfdf0cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:46.357875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.357850    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:46.359407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.359379    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz" pod "17153e24-fc3f-4582-b071-68db4cfdf0cf" (UID: "17153e24-fc3f-4582-b071-68db4cfdf0cf"). InnerVolumeSpecName "kube-api-access-v86xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:46.359504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.359408    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca" pod "17153e24-fc3f-4582-b071-68db4cfdf0cf" (UID: "17153e24-fc3f-4582-b071-68db4cfdf0cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:46.458772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.458741    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17153e24-fc3f-4582-b071-68db4cfdf0cf-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:46.458772 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:46.458769    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86xz\" (UniqueName: \"kubernetes.io/projected/17153e24-fc3f-4582-b071-68db4cfdf0cf-kube-api-access-v86xz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:47.082427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:47.082388    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"
Oct 31 01:53:47.082427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:47.082419    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh" event={"ID":"17153e24-fc3f-4582-b071-68db4cfdf0cf","Type":"ContainerDied","Data":"7020cde0ab9e4a6a16033da94d73f495ed0e69c460b78827ecc72f98e59eee26"}
Oct 31 01:53:47.082650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:47.082468    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7020cde0ab9e4a6a16033da94d73f495ed0e69c460b78827ecc72f98e59eee26"
Oct 31 01:53:52.986890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:52.986855    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:53:54.782169 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.779090    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"]
Oct 31 01:53:54.794160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.790989    5419 cpu_manager.go:414] "containerMap: removing container" podUID="17153e24-fc3f-4582-b071-68db4cfdf0cf" containerName="revoke-keys"
Oct 31 01:53:54.794160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.791022    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17153e24-fc3f-4582-b071-68db4cfdf0cf" containerName="revoke-keys"
Oct 31 01:53:54.794160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.791271    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="17153e24-fc3f-4582-b071-68db4cfdf0cf" containerName="revoke-keys"
Oct 31 01:53:54.796511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.796487    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:54.866262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.866221    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:53:54.903767 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.903735    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:53:54.982230 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.906341    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:53:54.982466 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.906477    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:53:54.986196 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.906572    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:53:54.992204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.969628    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:54.992490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.992469    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97cqc\" (UniqueName: \"kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:54.992823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:54.992790    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.045289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.045208    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"]
Oct 31 01:53:55.102897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.102859    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97cqc\" (UniqueName: \"kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.103283 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.103254    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.103654 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.103630    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.105866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.105839    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.137581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.137548    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.177080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.177041    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97cqc\" (UniqueName: \"kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc\") pod \"maas-api-revoke-sub-f6d32f34-cslrx\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.204532 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.204494    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:55.998479 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:55.993546    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"]
Oct 31 01:53:56.016159 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:56.014639    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93bde77_e0be_4cf5_85fa_4c1b8c097f42.slice/crio-78ae3c4c22e1f61253a742403575137f95965442681b2548d602c8f03dbf3ac3 WatchSource:0}: Error finding container 78ae3c4c22e1f61253a742403575137f95965442681b2548d602c8f03dbf3ac3: Status 404 returned error can't find the container with id 78ae3c4c22e1f61253a742403575137f95965442681b2548d602c8f03dbf3ac3
Oct 31 01:53:56.191593 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.191552    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx" event={"ID":"a93bde77-e0be-4cf5-85fa-4c1b8c097f42","Type":"ContainerStarted","Data":"78ae3c4c22e1f61253a742403575137f95965442681b2548d602c8f03dbf3ac3"}
Oct 31 01:53:56.630602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.630560    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"]
Oct 31 01:53:56.634787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.634763    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.653043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.652985    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.653168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.653039    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.653242 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.653228    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwl8\" (UniqueName: \"kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.711058 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.711022    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"]
Oct 31 01:53:56.754699 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.754667    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.754865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.754767    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwl8\" (UniqueName: \"kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.754865 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.754819    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.755336 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.755318    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.757377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.757344    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.811053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.811025    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwl8\" (UniqueName: \"kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8\") pod \"maas-api-revoke-sub-1d3eaeb2-ttwhk\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:56.948266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:56.948187    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:53:57.208076 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:57.207988    5419 generic.go:356] "Generic (PLEG): container finished" podID="a93bde77-e0be-4cf5-85fa-4c1b8c097f42" containerID="312e79fe120e8f31d18b0f99d04f8aef75598b9670c2e43d1b9f2044112a3238" exitCode=0
Oct 31 01:53:57.208457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:57.208076    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx" event={"ID":"a93bde77-e0be-4cf5-85fa-4c1b8c097f42","Type":"ContainerDied","Data":"312e79fe120e8f31d18b0f99d04f8aef75598b9670c2e43d1b9f2044112a3238"}
Oct 31 01:53:57.348289 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:57.348260    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e76f6a_b47a_43f3_8c93_583d8b3e45d9.slice/crio-e8ceafeca1ea9e648927b440ef76d064a9a096b545491d80af2ea86900e98846 WatchSource:0}: Error finding container e8ceafeca1ea9e648927b440ef76d064a9a096b545491d80af2ea86900e98846: Status 404 returned error can't find the container with id e8ceafeca1ea9e648927b440ef76d064a9a096b545491d80af2ea86900e98846
Oct 31 01:53:57.372598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:57.372571    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"]
Oct 31 01:53:58.062896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.062872    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:53:58.170841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.170757    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume\") pod \"3556719c-b6d4-4085-b84b-e88db0c6970a\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") "
Oct 31 01:53:58.170841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.170806    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29\") pod \"3556719c-b6d4-4085-b84b-e88db0c6970a\" (UID: \"3556719c-b6d4-4085-b84b-e88db0c6970a\") "
Oct 31 01:53:58.171130 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.171108    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume" pod "3556719c-b6d4-4085-b84b-e88db0c6970a" (UID: "3556719c-b6d4-4085-b84b-e88db0c6970a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:58.173114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.173092    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29" pod "3556719c-b6d4-4085-b84b-e88db0c6970a" (UID: "3556719c-b6d4-4085-b84b-e88db0c6970a"). InnerVolumeSpecName "kube-api-access-jnx29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:58.217929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.217886    5419 generic.go:356] "Generic (PLEG): container finished" podID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerID="61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528" exitCode=0
Oct 31 01:53:58.217929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.217920    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" event={"ID":"3556719c-b6d4-4085-b84b-e88db0c6970a","Type":"ContainerDied","Data":"61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528"}
Oct 31 01:53:58.218519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.217975    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx" event={"ID":"3556719c-b6d4-4085-b84b-e88db0c6970a","Type":"ContainerDied","Data":"b75019e5641335dbc354735c87a88349c6659f76766d2c512bd80f1ebca5da5b"}
Oct 31 01:53:58.218519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.217991    5419 scope.go:122] "RemoveContainer" containerID="61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528"
Oct 31 01:53:58.218519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.217929    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"
Oct 31 01:53:58.220261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.220233    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk" event={"ID":"75e76f6a-b47a-43f3-8c93-583d8b3e45d9","Type":"ContainerStarted","Data":"8e25a584707ae2523efabbd0e02226761525fd8625f748350179a5c737d5522e"}
Oct 31 01:53:58.220464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.220297    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk" event={"ID":"75e76f6a-b47a-43f3-8c93-583d8b3e45d9","Type":"ContainerStarted","Data":"e8ceafeca1ea9e648927b440ef76d064a9a096b545491d80af2ea86900e98846"}
Oct 31 01:53:58.239881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.239856    5419 scope.go:122] "RemoveContainer" containerID="61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528"
Oct 31 01:53:58.240254 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:53:58.240233    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528\": container with ID starting with 61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528 not found: ID does not exist" containerID="61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528"
Oct 31 01:53:58.240325 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.240261    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528"} err="failed to get container status \"61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528\": rpc error: code = NotFound desc = could not find container \"61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528\": container with ID starting with 61750ef05da20dafa1d38e7e11a2bf312f36731346b7be165c0c2d763dfa1528 not found: ID does not exist"
Oct 31 01:53:58.272421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.272390    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3556719c-b6d4-4085-b84b-e88db0c6970a-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:58.272569 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.272424    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnx29\" (UniqueName: \"kubernetes.io/projected/3556719c-b6d4-4085-b84b-e88db0c6970a-kube-api-access-jnx29\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:58.402597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.402566    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"]
Oct 31 01:53:58.406319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.406289    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c0daf4-77876767f-gc2dx"]
Oct 31 01:53:58.417920 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.417897    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:58.488172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.487506    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc\" (UniqueName: \"kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc\") pod \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") "
Oct 31 01:53:58.488172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.487593    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca\") pod \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") "
Oct 31 01:53:58.488172 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.487655    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca\") pod \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\" (UID: \"a93bde77-e0be-4cf5-85fa-4c1b8c097f42\") "
Oct 31 01:53:58.499993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.499697    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca" pod "a93bde77-e0be-4cf5-85fa-4c1b8c097f42" (UID: "a93bde77-e0be-4cf5-85fa-4c1b8c097f42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:53:58.500185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.500004    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca" pod "a93bde77-e0be-4cf5-85fa-4c1b8c097f42" (UID: "a93bde77-e0be-4cf5-85fa-4c1b8c097f42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:53:58.502580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.502552    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc" pod "a93bde77-e0be-4cf5-85fa-4c1b8c097f42" (UID: "a93bde77-e0be-4cf5-85fa-4c1b8c097f42"). InnerVolumeSpecName "kube-api-access-97cqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:53:58.503940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.503898    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk" podStartSLOduration=2.503885232 podStartE2EDuration="2.503885232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:58.499651842 +0000 UTC m=+2959.800608421" watchObservedRunningTime="2026-10-31 01:53:58.503885232 +0000 UTC m=+2959.804841866"
Oct 31 01:53:58.597164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.593549    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97cqc\" (UniqueName: \"kubernetes.io/projected/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-kube-api-access-97cqc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:58.597164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.593583    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:58.597164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.593601    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93bde77-e0be-4cf5-85fa-4c1b8c097f42-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:53:58.837714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:58.837639    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" path="/var/lib/kubelet/pods/3556719c-b6d4-4085-b84b-e88db0c6970a/volumes"
Oct 31 01:53:59.202032 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.201990    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"]
Oct 31 01:53:59.202583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.202566    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a93bde77-e0be-4cf5-85fa-4c1b8c097f42" containerName="revoke-keys"
Oct 31 01:53:59.202583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.202582    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93bde77-e0be-4cf5-85fa-4c1b8c097f42" containerName="revoke-keys"
Oct 31 01:53:59.202752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.202598    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerName="payload-processing"
Oct 31 01:53:59.202752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.202604    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerName="payload-processing"
Oct 31 01:53:59.202752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.202705    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93bde77-e0be-4cf5-85fa-4c1b8c097f42" containerName="revoke-keys"
Oct 31 01:53:59.202752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.202714    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3556719c-b6d4-4085-b84b-e88db0c6970a" containerName="payload-processing"
Oct 31 01:53:59.206418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.206397    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.228782 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.228750    5419 generic.go:356] "Generic (PLEG): container finished" podID="75e76f6a-b47a-43f3-8c93-583d8b3e45d9" containerID="8e25a584707ae2523efabbd0e02226761525fd8625f748350179a5c737d5522e" exitCode=0
Oct 31 01:53:59.229129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.228791    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk" event={"ID":"75e76f6a-b47a-43f3-8c93-583d8b3e45d9","Type":"ContainerDied","Data":"8e25a584707ae2523efabbd0e02226761525fd8625f748350179a5c737d5522e"}
Oct 31 01:53:59.230291 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.230264    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"
Oct 31 01:53:59.230398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.230297    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx" event={"ID":"a93bde77-e0be-4cf5-85fa-4c1b8c097f42","Type":"ContainerDied","Data":"78ae3c4c22e1f61253a742403575137f95965442681b2548d602c8f03dbf3ac3"}
Oct 31 01:53:59.230398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.230340    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78ae3c4c22e1f61253a742403575137f95965442681b2548d602c8f03dbf3ac3"
Oct 31 01:53:59.245951 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.245922    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"]
Oct 31 01:53:59.307344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.307086    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.307344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.307126    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rtt\" (UniqueName: \"kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.307344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.307234    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.407926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.407835    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.407926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.407878    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rtt\" (UniqueName: \"kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.407926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.407917    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.408783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.408745    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.410377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.410356    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.465181 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.465151    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rtt\" (UniqueName: \"kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt\") pod \"maas-api-revoke-sub-4191a066-xwkn6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.519927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.519895    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:53:59.916422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:53:59.916374    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"]
Oct 31 01:53:59.916859 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:53:59.916832    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8a23e0_2780_4c52_ad0b_0a7ad66c68a6.slice/crio-8452765a264c1de4bd5c37f6fbce029a1c343957482f73e574d879cfff86c9bd WatchSource:0}: Error finding container 8452765a264c1de4bd5c37f6fbce029a1c343957482f73e574d879cfff86c9bd: Status 404 returned error can't find the container with id 8452765a264c1de4bd5c37f6fbce029a1c343957482f73e574d879cfff86c9bd
Oct 31 01:54:00.238650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.238613    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6" event={"ID":"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6","Type":"ContainerStarted","Data":"8452765a264c1de4bd5c37f6fbce029a1c343957482f73e574d879cfff86c9bd"}
Oct 31 01:54:00.376991 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.376959    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:54:00.423768 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.423734    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca\") pod \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") "
Oct 31 01:54:00.423959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.423849    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8\" (UniqueName: \"kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8\") pod \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") "
Oct 31 01:54:00.423959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.423895    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca\") pod \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\" (UID: \"75e76f6a-b47a-43f3-8c93-583d8b3e45d9\") "
Oct 31 01:54:00.424093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.424073    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca" pod "75e76f6a-b47a-43f3-8c93-583d8b3e45d9" (UID: "75e76f6a-b47a-43f3-8c93-583d8b3e45d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:00.424350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.424332    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:00.426192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.426159    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca" pod "75e76f6a-b47a-43f3-8c93-583d8b3e45d9" (UID: "75e76f6a-b47a-43f3-8c93-583d8b3e45d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:00.426317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.426193    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8" pod "75e76f6a-b47a-43f3-8c93-583d8b3e45d9" (UID: "75e76f6a-b47a-43f3-8c93-583d8b3e45d9"). InnerVolumeSpecName "kube-api-access-nwwl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:00.525648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.525616    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwl8\" (UniqueName: \"kubernetes.io/projected/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-kube-api-access-nwwl8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:00.525648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:00.525648    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e76f6a-b47a-43f3-8c93-583d8b3e45d9-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:01.247889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:01.247860    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"
Oct 31 01:54:01.248333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:01.247861    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk" event={"ID":"75e76f6a-b47a-43f3-8c93-583d8b3e45d9","Type":"ContainerDied","Data":"e8ceafeca1ea9e648927b440ef76d064a9a096b545491d80af2ea86900e98846"}
Oct 31 01:54:01.248333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:01.247992    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8ceafeca1ea9e648927b440ef76d064a9a096b545491d80af2ea86900e98846"
Oct 31 01:54:01.250045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:01.250017    5419 generic.go:356] "Generic (PLEG): container finished" podID="bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" containerID="e4cf0df6e22183f89b2b8c747f57175ffb8a683e9544e04b9854bd45edf67696" exitCode=0
Oct 31 01:54:01.250369 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:01.250344    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6" event={"ID":"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6","Type":"ContainerDied","Data":"e4cf0df6e22183f89b2b8c747f57175ffb8a683e9544e04b9854bd45edf67696"}
Oct 31 01:54:02.413703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.413679    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:54:02.450222 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.450189    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca\") pod \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") "
Oct 31 01:54:02.450392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.450250    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca\") pod \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") "
Oct 31 01:54:02.450457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.450390    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt\" (UniqueName: \"kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt\") pod \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\" (UID: \"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6\") "
Oct 31 01:54:02.450640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.450614    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca" pod "bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" (UID: "bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:02.450886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.450869    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:02.453085 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.453059    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca" pod "bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" (UID: "bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:02.453245 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.453089    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt" pod "bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" (UID: "bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6"). InnerVolumeSpecName "kube-api-access-t9rtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:02.552682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.552612    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9rtt\" (UniqueName: \"kubernetes.io/projected/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-kube-api-access-t9rtt\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:02.552682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:02.552644    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:03.265204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:03.265170    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"
Oct 31 01:54:03.265204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:03.265188    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6" event={"ID":"bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6","Type":"ContainerDied","Data":"8452765a264c1de4bd5c37f6fbce029a1c343957482f73e574d879cfff86c9bd"}
Oct 31 01:54:03.265473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:03.265238    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8452765a264c1de4bd5c37f6fbce029a1c343957482f73e574d879cfff86c9bd"
Oct 31 01:54:04.273471 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:04.273429    5419 generic.go:356] "Generic (PLEG): container finished" podID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerID="b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93" exitCode=0
Oct 31 01:54:04.273906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:04.273500    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" event={"ID":"f9a9a077-da45-490a-8148-6069b2f37d0c","Type":"ContainerDied","Data":"b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"}
Oct 31 01:54:04.273906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:04.273889    5419 scope.go:122] "RemoveContainer" containerID="b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"
Oct 31 01:54:04.275302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:04.275275    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:04.275380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:04.275354    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:04.275425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:04.275389    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:05.305318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.305279    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" event={"ID":"f9a9a077-da45-490a-8148-6069b2f37d0c","Type":"ContainerStarted","Data":"3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949"}
Oct 31 01:54:05.524714 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.524683    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"]
Oct 31 01:54:05.525210 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.525195    5419 cpu_manager.go:414] "containerMap: removing container" podUID="bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" containerName="revoke-keys"
Oct 31 01:54:05.525268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.525210    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" containerName="revoke-keys"
Oct 31 01:54:05.525268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.525241    5419 cpu_manager.go:414] "containerMap: removing container" podUID="75e76f6a-b47a-43f3-8c93-583d8b3e45d9" containerName="revoke-keys"
Oct 31 01:54:05.525268 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.525246    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e76f6a-b47a-43f3-8c93-583d8b3e45d9" containerName="revoke-keys"
Oct 31 01:54:05.525370 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.525341    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" containerName="revoke-keys"
Oct 31 01:54:05.525370 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.525354    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e76f6a-b47a-43f3-8c93-583d8b3e45d9" containerName="revoke-keys"
Oct 31 01:54:05.528960 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.528942    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.548500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.548472    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"]
Oct 31 01:54:05.599431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.599339    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:05.599630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.599368    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:05.599630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.599425    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:05.599630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.599522    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:54:05.604119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.604087    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.604270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.604152    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.604270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.604257    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-558zp\" (UniqueName: \"kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.705599 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.705561    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.705785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.705621    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.705785 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.705668    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-558zp\" (UniqueName: \"kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.706219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.706197    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.708262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.708237    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.764913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.764858    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-558zp\" (UniqueName: \"kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp\") pod \"maas-api-revoke-sub-bacb09ee-r6l99\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:05.841908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:05.841875    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:06.003486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:06.003403    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:54:06.004384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:06.004341    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:54:06.004501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:06.004385    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:54:06.249252 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:06.249214    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"]
Oct 31 01:54:06.314485 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:06.314456    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99" event={"ID":"d48bca69-e103-43f6-9757-fe8a87f1b90d","Type":"ContainerStarted","Data":"af4201bcb272a7b5a760d24d3bed40d7aa07ff07d82a3c6ee668094a46253d20"}
Oct 31 01:54:07.005180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:07.004106    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:54:07.005180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:07.004183    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:54:07.323303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:07.323210    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99" event={"ID":"d48bca69-e103-43f6-9757-fe8a87f1b90d","Type":"ContainerStarted","Data":"5ee1be29562a443afe0ad57f827fe49ac2588a55b45ee6c83fbc0bcdda725030"}
Oct 31 01:54:07.446990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:07.446937    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99" podStartSLOduration=2.446918433 podStartE2EDuration="2.446918433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:07.444655009 +0000 UTC m=+2968.745611586" watchObservedRunningTime="2026-10-31 01:54:07.446918433 +0000 UTC m=+2968.747875015"
Oct 31 01:54:07.456455 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:07.456433    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:54:08.003480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.003441    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:54:08.003717 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.003491    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:54:08.044744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.044697    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"]
Oct 31 01:54:08.047710 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.047680    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba569dd-fqw2t"]
Oct 31 01:54:08.331748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.331655    5419 generic.go:356] "Generic (PLEG): container finished" podID="d48bca69-e103-43f6-9757-fe8a87f1b90d" containerID="5ee1be29562a443afe0ad57f827fe49ac2588a55b45ee6c83fbc0bcdda725030" exitCode=0
Oct 31 01:54:08.331748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.331703    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99" event={"ID":"d48bca69-e103-43f6-9757-fe8a87f1b90d","Type":"ContainerDied","Data":"5ee1be29562a443afe0ad57f827fe49ac2588a55b45ee6c83fbc0bcdda725030"}
Oct 31 01:54:08.503426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.503389    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"]
Oct 31 01:54:08.508256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.508228    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.606559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.606480    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"]
Oct 31 01:54:08.642580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.642539    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.642763 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.642624    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.642820 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.642791    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.642964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.642937    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.643014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.642996    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.643051 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.643031    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.643099 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.643086    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.643153 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.643108    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.643213 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.643201    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9k59\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744449    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744520    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744547    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744562    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744581    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744596    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744637    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9k59\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.744688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744679    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.745089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.744735    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.745089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.745014    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.745089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.745045    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.745089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.745068    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.745358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.745106    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.745619 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.745599    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.746926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.746908    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.747696 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.747674    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.802346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.802304    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9k59\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.802742 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.802719    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token\") pod \"e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") " pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.822996 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.822968    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:08.839744 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:08.839338    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e" path="/var/lib/kubelet/pods/04ab0ad8-32ba-4b81-8ec9-6ba7a48d968e/volumes"
Oct 31 01:54:09.004310 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.004262    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:54:09.004522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.004324    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:54:09.269377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.269340    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"]
Oct 31 01:54:09.269769 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:09.269744    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7faebc63_2f55_4fc6_b159_2a462ecdffd2.slice/crio-145acac5b051bfd5e53356437bd075c7580b26c5b5ea6772bd77f28d13bde357 WatchSource:0}: Error finding container 145acac5b051bfd5e53356437bd075c7580b26c5b5ea6772bd77f28d13bde357: Status 404 returned error can't find the container with id 145acac5b051bfd5e53356437bd075c7580b26c5b5ea6772bd77f28d13bde357
Oct 31 01:54:09.272205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.272174    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:09.272341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.272248    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:09.272341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.272293    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:09.340613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.340585    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" event={"ID":"7faebc63-2f55-4fc6-b159-2a462ecdffd2","Type":"ContainerStarted","Data":"145acac5b051bfd5e53356437bd075c7580b26c5b5ea6772bd77f28d13bde357"}
Oct 31 01:54:09.524571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.524503    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:09.655949 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.655915    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp\" (UniqueName: \"kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp\") pod \"d48bca69-e103-43f6-9757-fe8a87f1b90d\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") "
Oct 31 01:54:09.656188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.655985    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca\") pod \"d48bca69-e103-43f6-9757-fe8a87f1b90d\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") "
Oct 31 01:54:09.656188 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.656174    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca\") pod \"d48bca69-e103-43f6-9757-fe8a87f1b90d\" (UID: \"d48bca69-e103-43f6-9757-fe8a87f1b90d\") "
Oct 31 01:54:09.656813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.656785    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca" pod "d48bca69-e103-43f6-9757-fe8a87f1b90d" (UID: "d48bca69-e103-43f6-9757-fe8a87f1b90d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:09.658404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.658378    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca" pod "d48bca69-e103-43f6-9757-fe8a87f1b90d" (UID: "d48bca69-e103-43f6-9757-fe8a87f1b90d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:09.658523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.658414    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp" pod "d48bca69-e103-43f6-9757-fe8a87f1b90d" (UID: "d48bca69-e103-43f6-9757-fe8a87f1b90d"). InnerVolumeSpecName "kube-api-access-558zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:09.760750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.760654    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.760750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.760697    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-558zp\" (UniqueName: \"kubernetes.io/projected/d48bca69-e103-43f6-9757-fe8a87f1b90d-kube-api-access-558zp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:09.760750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:09.760715    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48bca69-e103-43f6-9757-fe8a87f1b90d-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:10.003746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.003706    5419 patch_prober.go:29] interesting pod/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body=
Oct 31 01:54:10.003940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.003781    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused"
Oct 31 01:54:10.349705 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.349606    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" event={"ID":"7faebc63-2f55-4fc6-b159-2a462ecdffd2","Type":"ContainerStarted","Data":"58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304"}
Oct 31 01:54:10.351165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.351111    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99" event={"ID":"d48bca69-e103-43f6-9757-fe8a87f1b90d","Type":"ContainerDied","Data":"af4201bcb272a7b5a760d24d3bed40d7aa07ff07d82a3c6ee668094a46253d20"}
Oct 31 01:54:10.351316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.351204    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4201bcb272a7b5a760d24d3bed40d7aa07ff07d82a3c6ee668094a46253d20"
Oct 31 01:54:10.351316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.351233    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"
Oct 31 01:54:10.526314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.526265    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podStartSLOduration=2.526251156 podStartE2EDuration="2.526251156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:10.519183331 +0000 UTC m=+2971.820139932" watchObservedRunningTime="2026-10-31 01:54:10.526251156 +0000 UTC m=+2971.827207733"
Oct 31 01:54:10.823241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.823209    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:10.824889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.824866    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 31 01:54:10.825003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:10.824922    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 31 01:54:11.008845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:11.008818    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:54:11.010048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:11.010022    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:54:11.823400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:11.823359    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 31 01:54:11.823860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:11.823434    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 31 01:54:12.824039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:12.824004    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 31 01:54:12.824548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:12.824066    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 31 01:54:13.824092 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:13.824050    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 31 01:54:13.824741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:13.824709    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 31 01:54:14.635766 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:54:14.635724    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:54:14.828618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:14.828574    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 31 01:54:14.829018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:14.828641    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 31 01:54:15.823315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:15.823280    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Oct 31 01:54:15.823495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:15.823340    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Oct 31 01:54:16.779614 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.779575    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"]
Oct 31 01:54:16.780373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.780350    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d48bca69-e103-43f6-9757-fe8a87f1b90d" containerName="revoke-keys"
Oct 31 01:54:16.780492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.780373    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48bca69-e103-43f6-9757-fe8a87f1b90d" containerName="revoke-keys"
Oct 31 01:54:16.780554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.780529    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48bca69-e103-43f6-9757-fe8a87f1b90d" containerName="revoke-keys"
Oct 31 01:54:16.794119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.794091    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:16.843700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.843672    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"]
Oct 31 01:54:16.889385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.889356    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:16.889385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.889372    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:16.889603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.889405    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:54:16.895108 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.895087    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:16.924592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.924564    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:16.924738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.924605    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqszz\" (UniqueName: \"kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:16.924791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:16.924767    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.025683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.025648    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.025683 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.025687    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqszz\" (UniqueName: \"kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.025923 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.025760    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.026461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.026428    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.028475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.028446    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.082758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.082672    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqszz\" (UniqueName: \"kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz\") pod \"maas-api-revoke-sub-43c0d396-fxcts\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.108554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.108522    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:17.489638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.489604    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"]
Oct 31 01:54:17.490967 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:17.490933    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe41142d_f8dc_4241_8378_df71319b8a4e.slice/crio-e79fe3e2ff8cba68c7a8a651fb999e727436521111e7eb723bf6d6a677fd0f4c WatchSource:0}: Error finding container e79fe3e2ff8cba68c7a8a651fb999e727436521111e7eb723bf6d6a677fd0f4c: Status 404 returned error can't find the container with id e79fe3e2ff8cba68c7a8a651fb999e727436521111e7eb723bf6d6a677fd0f4c
Oct 31 01:54:17.864314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.864273    5419 patch_prober.go:29] interesting pod/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:54:17.864706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:17.864322    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:18.150026 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:18.149930    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:18.150700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:18.150600    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:18.152109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:18.152085    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:54:18.485419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:18.485352    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts" event={"ID":"be41142d-f8dc-4241-8378-df71319b8a4e","Type":"ContainerStarted","Data":"b78d1594875726bc5a587516efc46649cd62a78c506bffae26d0ffdc6cbcc44d"}
Oct 31 01:54:18.485571 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:18.485479    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts" event={"ID":"be41142d-f8dc-4241-8378-df71319b8a4e","Type":"ContainerStarted","Data":"e79fe3e2ff8cba68c7a8a651fb999e727436521111e7eb723bf6d6a677fd0f4c"}
Oct 31 01:54:18.600050 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:18.600004    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts" podStartSLOduration=2.599990379 podStartE2EDuration="2.599990379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:18.599326506 +0000 UTC m=+2979.900283096" watchObservedRunningTime="2026-10-31 01:54:18.599990379 +0000 UTC m=+2979.900946955"
Oct 31 01:54:19.494208 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:19.494170    5419 generic.go:356] "Generic (PLEG): container finished" podID="be41142d-f8dc-4241-8378-df71319b8a4e" containerID="b78d1594875726bc5a587516efc46649cd62a78c506bffae26d0ffdc6cbcc44d" exitCode=0
Oct 31 01:54:19.495254 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:19.495217    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts" event={"ID":"be41142d-f8dc-4241-8378-df71319b8a4e","Type":"ContainerDied","Data":"b78d1594875726bc5a587516efc46649cd62a78c506bffae26d0ffdc6cbcc44d"}
Oct 31 01:54:19.936773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:19.936693    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"]
Oct 31 01:54:19.940940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:19.940913    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.047394 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.047358    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"]
Oct 31 01:54:20.064071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064029    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064071 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064068    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064108    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064173    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrj5\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064273    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064376    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064498    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064580 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064550    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.064757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.064606    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.087257 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.087232    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057e2d-openshift-default-dockercfg-2bvxk\""
Oct 31 01:54:20.166021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.165978    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166020    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166056    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166078    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166095    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166236    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166287 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166282    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrj5\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166322    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166422    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166534    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166559    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166650    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166776    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.166878 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.166867    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.168540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.168512    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.169087 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.169065    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.225298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.225263    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.225443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.225356    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrj5\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.255368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.255333    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:20.640371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.640346    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:20.686229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.686195    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"]
Oct 31 01:54:20.687745 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:20.687715    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069362af_682c_4fd6_af88_10532a5e459d.slice/crio-e79098cc59b7fbad15f300b6545f87c402a4fd57a3999c4c6764f2f49ae35616 WatchSource:0}: Error finding container e79098cc59b7fbad15f300b6545f87c402a4fd57a3999c4c6764f2f49ae35616: Status 404 returned error can't find the container with id e79098cc59b7fbad15f300b6545f87c402a4fd57a3999c4c6764f2f49ae35616
Oct 31 01:54:20.690837 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.690799    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:20.690964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.690886    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:20.690964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.690933    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:20.775170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.775127    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca\") pod \"be41142d-f8dc-4241-8378-df71319b8a4e\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") "
Oct 31 01:54:20.775570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.775244    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca\") pod \"be41142d-f8dc-4241-8378-df71319b8a4e\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") "
Oct 31 01:54:20.775570 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.775324    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz\" (UniqueName: \"kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz\") pod \"be41142d-f8dc-4241-8378-df71319b8a4e\" (UID: \"be41142d-f8dc-4241-8378-df71319b8a4e\") "
Oct 31 01:54:20.775733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.775677    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca" pod "be41142d-f8dc-4241-8378-df71319b8a4e" (UID: "be41142d-f8dc-4241-8378-df71319b8a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:20.775832 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.775811    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:20.777842 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.777822    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca" pod "be41142d-f8dc-4241-8378-df71319b8a4e" (UID: "be41142d-f8dc-4241-8378-df71319b8a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:20.778069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.778042    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz" pod "be41142d-f8dc-4241-8378-df71319b8a4e" (UID: "be41142d-f8dc-4241-8378-df71319b8a4e"). InnerVolumeSpecName "kube-api-access-sqszz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:20.877175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.877112    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be41142d-f8dc-4241-8378-df71319b8a4e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:20.877175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:20.877152    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqszz\" (UniqueName: \"kubernetes.io/projected/be41142d-f8dc-4241-8378-df71319b8a4e-kube-api-access-sqszz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:21.221625 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.221589    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"]
Oct 31 01:54:21.222178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.222155    5419 cpu_manager.go:414] "containerMap: removing container" podUID="be41142d-f8dc-4241-8378-df71319b8a4e" containerName="revoke-keys"
Oct 31 01:54:21.222178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.222175    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be41142d-f8dc-4241-8378-df71319b8a4e" containerName="revoke-keys"
Oct 31 01:54:21.222385 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.222308    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="be41142d-f8dc-4241-8378-df71319b8a4e" containerName="revoke-keys"
Oct 31 01:54:21.226301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.226282    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.282680 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.282604    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"]
Oct 31 01:54:21.392974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.392932    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.393344 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.393316    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.393598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.393581    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7frg\" (UniqueName: \"kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.500873 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.500826    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7frg\" (UniqueName: \"kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.501112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.500970    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.501112 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.501022    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.501786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.501758    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.505270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.504614    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.510795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.510773    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"
Oct 31 01:54:21.510795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.510777    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts" event={"ID":"be41142d-f8dc-4241-8378-df71319b8a4e","Type":"ContainerDied","Data":"e79fe3e2ff8cba68c7a8a651fb999e727436521111e7eb723bf6d6a677fd0f4c"}
Oct 31 01:54:21.510971 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.510832    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e79fe3e2ff8cba68c7a8a651fb999e727436521111e7eb723bf6d6a677fd0f4c"
Oct 31 01:54:21.512567 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.512538    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" event={"ID":"069362af-682c-4fd6-af88-10532a5e459d","Type":"ContainerStarted","Data":"29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b"}
Oct 31 01:54:21.512652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.512576    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" event={"ID":"069362af-682c-4fd6-af88-10532a5e459d","Type":"ContainerStarted","Data":"e79098cc59b7fbad15f300b6545f87c402a4fd57a3999c4c6764f2f49ae35616"}
Oct 31 01:54:21.557610 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.557566    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7frg\" (UniqueName: \"kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg\") pod \"maas-api-revoke-sub-df862bdf-lcp6v\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.632632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.632579    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podStartSLOduration=2.632564335 podStartE2EDuration="2.632564335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:21.628965061 +0000 UTC m=+2982.929921640" watchObservedRunningTime="2026-10-31 01:54:21.632564335 +0000 UTC m=+2982.933520911"
Oct 31 01:54:21.841151 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.841052    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:21.860187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.860152    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"]
Oct 31 01:54:21.865503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.865483    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:21.924123 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:21.924031    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"]
Oct 31 01:54:22.007533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.007500    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvtl\" (UniqueName: \"kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.007712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.007638    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.007893 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.007864    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.109270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.109181    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.109270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.109243    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvtl\" (UniqueName: \"kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.109465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.109298    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.109887 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.109867    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.111908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.111885    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.165843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.165811    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvtl\" (UniqueName: \"kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl\") pod \"maas-api-revoke-sub-69d2a4a5-srrsh\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.215678 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.215644    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:22.247043 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:22.247011    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6bf222_b732_48e6_a49a_e28d7ee979c5.slice/crio-a07ff39290191623fb867cac91c1844fa64ba38a4b198e2d9635d811bb75ab96 WatchSource:0}: Error finding container a07ff39290191623fb867cac91c1844fa64ba38a4b198e2d9635d811bb75ab96: Status 404 returned error can't find the container with id a07ff39290191623fb867cac91c1844fa64ba38a4b198e2d9635d811bb75ab96
Oct 31 01:54:22.250559 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.250534    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"]
Oct 31 01:54:22.256021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.255996    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:22.257890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.257865    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 31 01:54:22.257964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.257929    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 31 01:54:22.521629 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.521601    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v" event={"ID":"cc6bf222-b732-48e6-a49a-e28d7ee979c5","Type":"ContainerStarted","Data":"a07ff39290191623fb867cac91c1844fa64ba38a4b198e2d9635d811bb75ab96"}
Oct 31 01:54:22.623267 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:22.623245    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"]
Oct 31 01:54:22.624104 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:22.624077    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5e8b9e_ce60_4185_98e7_333dfac58b56.slice/crio-4b9eae1aa4c3d56cd015985b855ecfc3bd64a20e916b960d44a3fd442bf823df WatchSource:0}: Error finding container 4b9eae1aa4c3d56cd015985b855ecfc3bd64a20e916b960d44a3fd442bf823df: Status 404 returned error can't find the container with id 4b9eae1aa4c3d56cd015985b855ecfc3bd64a20e916b960d44a3fd442bf823df
Oct 31 01:54:23.256384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.256342    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 31 01:54:23.256759 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.256414    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 31 01:54:23.530270 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.530178    5419 generic.go:356] "Generic (PLEG): container finished" podID="cc6bf222-b732-48e6-a49a-e28d7ee979c5" containerID="757b3605e20342090dfc70d40f1ce9fd6fd3064866c31c9c221bf15159cefc85" exitCode=0
Oct 31 01:54:23.530453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.530277    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v" event={"ID":"cc6bf222-b732-48e6-a49a-e28d7ee979c5","Type":"ContainerDied","Data":"757b3605e20342090dfc70d40f1ce9fd6fd3064866c31c9c221bf15159cefc85"}
Oct 31 01:54:23.531874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.531851    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh" event={"ID":"ba5e8b9e-ce60-4185-98e7-333dfac58b56","Type":"ContainerStarted","Data":"64ff47b52e6bc5fd71ef305c923096e56c48bb60571c523df2f47910f7d48374"}
Oct 31 01:54:23.531998 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.531893    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh" event={"ID":"ba5e8b9e-ce60-4185-98e7-333dfac58b56","Type":"ContainerStarted","Data":"4b9eae1aa4c3d56cd015985b855ecfc3bd64a20e916b960d44a3fd442bf823df"}
Oct 31 01:54:23.806079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:23.805505    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh" podStartSLOduration=2.805489145 podStartE2EDuration="2.805489145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:23.784779173 +0000 UTC m=+2985.085735793" watchObservedRunningTime="2026-10-31 01:54:23.805489145 +0000 UTC m=+2985.106445722"
Oct 31 01:54:24.256334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.256298    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 31 01:54:24.256526 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.256362    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 31 01:54:24.541349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.541271    5419 generic.go:356] "Generic (PLEG): container finished" podID="ba5e8b9e-ce60-4185-98e7-333dfac58b56" containerID="64ff47b52e6bc5fd71ef305c923096e56c48bb60571c523df2f47910f7d48374" exitCode=0
Oct 31 01:54:24.541506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.541361    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh" event={"ID":"ba5e8b9e-ce60-4185-98e7-333dfac58b56","Type":"ContainerDied","Data":"64ff47b52e6bc5fd71ef305c923096e56c48bb60571c523df2f47910f7d48374"}
Oct 31 01:54:24.732417 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.732391    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:24.867889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.867801    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca\") pod \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") "
Oct 31 01:54:24.867889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.867854    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg\" (UniqueName: \"kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg\") pod \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") "
Oct 31 01:54:24.868153 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.867975    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca\") pod \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\" (UID: \"cc6bf222-b732-48e6-a49a-e28d7ee979c5\") "
Oct 31 01:54:24.868235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.868130    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca" pod "cc6bf222-b732-48e6-a49a-e28d7ee979c5" (UID: "cc6bf222-b732-48e6-a49a-e28d7ee979c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:24.868598 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.868574    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:24.870244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.870216    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca" pod "cc6bf222-b732-48e6-a49a-e28d7ee979c5" (UID: "cc6bf222-b732-48e6-a49a-e28d7ee979c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:24.870382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.870276    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg" pod "cc6bf222-b732-48e6-a49a-e28d7ee979c5" (UID: "cc6bf222-b732-48e6-a49a-e28d7ee979c5"). InnerVolumeSpecName "kube-api-access-w7frg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:24.969318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.969287    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7frg\" (UniqueName: \"kubernetes.io/projected/cc6bf222-b732-48e6-a49a-e28d7ee979c5-kube-api-access-w7frg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:24.969318 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:24.969317    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6bf222-b732-48e6-a49a-e28d7ee979c5-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:25.256273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.256242    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 31 01:54:25.256449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.256304    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 31 01:54:25.549477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.549395    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"
Oct 31 01:54:25.549477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.549412    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v" event={"ID":"cc6bf222-b732-48e6-a49a-e28d7ee979c5","Type":"ContainerDied","Data":"a07ff39290191623fb867cac91c1844fa64ba38a4b198e2d9635d811bb75ab96"}
Oct 31 01:54:25.549477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.549456    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a07ff39290191623fb867cac91c1844fa64ba38a4b198e2d9635d811bb75ab96"
Oct 31 01:54:25.736886 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.736858    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:25.801681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.801587    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca\") pod \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") "
Oct 31 01:54:25.801861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.801682    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl\" (UniqueName: \"kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl\") pod \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") "
Oct 31 01:54:25.801861 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.801795    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca\") pod \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\" (UID: \"ba5e8b9e-ce60-4185-98e7-333dfac58b56\") "
Oct 31 01:54:25.802036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.802000    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca" pod "ba5e8b9e-ce60-4185-98e7-333dfac58b56" (UID: "ba5e8b9e-ce60-4185-98e7-333dfac58b56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:25.802253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.802235    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:25.804038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.804012    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca" pod "ba5e8b9e-ce60-4185-98e7-333dfac58b56" (UID: "ba5e8b9e-ce60-4185-98e7-333dfac58b56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:25.804159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.804035    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl" pod "ba5e8b9e-ce60-4185-98e7-333dfac58b56" (UID: "ba5e8b9e-ce60-4185-98e7-333dfac58b56"). InnerVolumeSpecName "kube-api-access-bnvtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:25.905903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.905863    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba5e8b9e-ce60-4185-98e7-333dfac58b56-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:25.905903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:25.905903    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvtl\" (UniqueName: \"kubernetes.io/projected/ba5e8b9e-ce60-4185-98e7-333dfac58b56-kube-api-access-bnvtl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:26.256356 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.256323    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 31 01:54:26.256561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.256391    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 31 01:54:26.532603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.532518    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"]
Oct 31 01:54:26.536422 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.536384    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cc6bf222-b732-48e6-a49a-e28d7ee979c5" containerName="revoke-keys"
Oct 31 01:54:26.536581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.536428    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6bf222-b732-48e6-a49a-e28d7ee979c5" containerName="revoke-keys"
Oct 31 01:54:26.536581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.536467    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ba5e8b9e-ce60-4185-98e7-333dfac58b56" containerName="revoke-keys"
Oct 31 01:54:26.536581 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.536475    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba5e8b9e-ce60-4185-98e7-333dfac58b56" containerName="revoke-keys"
Oct 31 01:54:26.536944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.536927    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6bf222-b732-48e6-a49a-e28d7ee979c5" containerName="revoke-keys"
Oct 31 01:54:26.537009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.536958    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba5e8b9e-ce60-4185-98e7-333dfac58b56" containerName="revoke-keys"
Oct 31 01:54:26.542170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.542126    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"]
Oct 31 01:54:26.546731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.546564    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.548277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.548256    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.562354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.558563    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh" event={"ID":"ba5e8b9e-ce60-4185-98e7-333dfac58b56","Type":"ContainerDied","Data":"4b9eae1aa4c3d56cd015985b855ecfc3bd64a20e916b960d44a3fd442bf823df"}
Oct 31 01:54:26.562354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.558609    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b9eae1aa4c3d56cd015985b855ecfc3bd64a20e916b960d44a3fd442bf823df"
Oct 31 01:54:26.562354 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.558573    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"
Oct 31 01:54:26.572721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.572670    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"]
Oct 31 01:54:26.616175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.611514    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"]
Oct 31 01:54:26.627941 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.627902    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.627966    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628020    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628081 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628053    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp6nx\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628107    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628205    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628370    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628463 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628409    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.628521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.628470    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.729592 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729561    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.729790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729610    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.729790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729655    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.729790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729724    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.729978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729826    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp6nx\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.729978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729889    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.729978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729923    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.729978 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.729959    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.730261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730003    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730023    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730117    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.730261 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730237    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730299    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730328    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730365    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.730490 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730462    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55w2\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.730734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730510    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730562    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730536    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.730734 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730697    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730815    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730853    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.730929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.730904    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.732880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.732853    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.733591 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.733570    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.787174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.787078    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.787329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.787230    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp6nx\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:26.832075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832037    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832109    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55w2\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832176    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832298 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832282    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832318    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832375    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832404    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832436    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.832495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832462    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.833565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832813    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.833565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.832833    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.833565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.833042    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.833565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.833281    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.833565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.833281    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.835190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.835168    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.835826 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.835802    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.888693 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.888657    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.888851 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.888763    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55w2\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2\") pod \"e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") " pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:26.918631 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:26.918604    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:27.165708 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.165618    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:27.256096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.256059    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 31 01:54:27.256278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.256120    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 31 01:54:27.356167 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.356125    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"]
Oct 31 01:54:27.356771 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:27.356749    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b31c94_8461_4c4d_88cd_9bdf60eeac63.slice/crio-cc01600c76eed00724c9460d6e3cb6cdf608563c0ec11a9ef69acbc6f46ee316 WatchSource:0}: Error finding container cc01600c76eed00724c9460d6e3cb6cdf608563c0ec11a9ef69acbc6f46ee316: Status 404 returned error can't find the container with id cc01600c76eed00724c9460d6e3cb6cdf608563c0ec11a9ef69acbc6f46ee316
Oct 31 01:54:27.359368 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.359334    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:27.359500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.359418    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:27.359500 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.359450    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:27.550390 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.550357    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"]
Oct 31 01:54:27.550758 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:27.550733    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba157891_0965_474c_af24_d4deabbfc812.slice/crio-a7be6496827521292ef54229ce47a32f6719cba617d048145013695a08cd2cc2 WatchSource:0}: Error finding container a7be6496827521292ef54229ce47a32f6719cba617d048145013695a08cd2cc2: Status 404 returned error can't find the container with id a7be6496827521292ef54229ce47a32f6719cba617d048145013695a08cd2cc2
Oct 31 01:54:27.553897 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.553867    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:27.553974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.553927    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:27.553974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.553959    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:27.567716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.567682    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"]
Oct 31 01:54:27.574464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.574430    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" event={"ID":"ba157891-0965-474c-af24-d4deabbfc812","Type":"ContainerStarted","Data":"a7be6496827521292ef54229ce47a32f6719cba617d048145013695a08cd2cc2"}
Oct 31 01:54:27.574638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.574619    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" event={"ID":"51b31c94-8461-4c4d-88cd-9bdf60eeac63","Type":"ContainerStarted","Data":"db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037"}
Oct 31 01:54:27.574787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.574532    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.574787 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.574728    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" event={"ID":"51b31c94-8461-4c4d-88cd-9bdf60eeac63","Type":"ContainerStarted","Data":"cc01600c76eed00724c9460d6e3cb6cdf608563c0ec11a9ef69acbc6f46ee316"}
Oct 31 01:54:27.626984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.626909    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"]
Oct 31 01:54:27.628017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.627992    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:27.628017 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.627992    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:27.628327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.628111    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:54:27.628950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.628931    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:27.642350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.641862    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.642350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.642193    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.642350 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.642242    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfpl\" (UniqueName: \"kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.701229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.698792    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podStartSLOduration=1.698774657 podStartE2EDuration="1.698774657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:27.694556448 +0000 UTC m=+2988.995513027" watchObservedRunningTime="2026-10-31 01:54:27.698774657 +0000 UTC m=+2988.999731235"
Oct 31 01:54:27.744278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.743403    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfpl\" (UniqueName: \"kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.744278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.743488    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.744278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.743704    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.744925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.744675    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.748315 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.748258    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.800097 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.799958    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfpl\" (UniqueName: \"kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl\") pod \"maas-api-revoke-sub-ca630013-c29ch\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.903165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.902670    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:27.924480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:27.924051    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:28.372049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:28.372019    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"]
Oct 31 01:54:28.373994 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:28.373965    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39f7ee8_b620_4450_8c06_ade4d275b19a.slice/crio-bf94a65b15d784f3bebe123a9d6457b10341a380a9d908bc4fac6e7a584195d9 WatchSource:0}: Error finding container bf94a65b15d784f3bebe123a9d6457b10341a380a9d908bc4fac6e7a584195d9: Status 404 returned error can't find the container with id bf94a65b15d784f3bebe123a9d6457b10341a380a9d908bc4fac6e7a584195d9
Oct 31 01:54:28.470809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:28.469128    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:28.470809 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:28.470663    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:28.472884 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:28.472834    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 01:54:28.970632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:28.970594    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 31 01:54:28.971232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:28.971178    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 31 01:54:29.160388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.159579    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" event={"ID":"ba157891-0965-474c-af24-d4deabbfc812","Type":"ContainerStarted","Data":"8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288"}
Oct 31 01:54:29.160388 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.159651    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch" event={"ID":"d39f7ee8-b620-4450-8c06-ade4d275b19a","Type":"ContainerStarted","Data":"bf94a65b15d784f3bebe123a9d6457b10341a380a9d908bc4fac6e7a584195d9"}
Oct 31 01:54:29.166415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.166390    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:29.170836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.170806    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 31 01:54:29.170955 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.170857    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 31 01:54:29.922163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.921706    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 31 01:54:29.922163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:29.921771    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 31 01:54:30.132231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.130966    5419 generic.go:356] "Generic (PLEG): container finished" podID="d39f7ee8-b620-4450-8c06-ade4d275b19a" containerID="81c5f98fc7f63cd64d632b0cbca9f32b6a9e5e0bd92d06e527f21a0499817445" exitCode=0
Oct 31 01:54:30.132231 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.131020    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch" event={"ID":"d39f7ee8-b620-4450-8c06-ade4d275b19a","Type":"ContainerDied","Data":"81c5f98fc7f63cd64d632b0cbca9f32b6a9e5e0bd92d06e527f21a0499817445"}
Oct 31 01:54:30.166914 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.166875    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 31 01:54:30.167117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.166926    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 31 01:54:30.260116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.260061    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podStartSLOduration=4.260047989 podStartE2EDuration="4.260047989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:29.315465929 +0000 UTC m=+2990.616422507" watchObservedRunningTime="2026-10-31 01:54:30.260047989 +0000 UTC m=+2991.561004568"
Oct 31 01:54:30.924164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.923127    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 31 01:54:30.924164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:30.923216    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 31 01:54:31.167170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.166867    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 31 01:54:31.167170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.166917    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 31 01:54:31.434719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.434693    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:31.553896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.553812    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca\") pod \"d39f7ee8-b620-4450-8c06-ade4d275b19a\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") "
Oct 31 01:54:31.553896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.553870    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca\") pod \"d39f7ee8-b620-4450-8c06-ade4d275b19a\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") "
Oct 31 01:54:31.554170 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.553929    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl\" (UniqueName: \"kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl\") pod \"d39f7ee8-b620-4450-8c06-ade4d275b19a\" (UID: \"d39f7ee8-b620-4450-8c06-ade4d275b19a\") "
Oct 31 01:54:31.554253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.554162    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca" pod "d39f7ee8-b620-4450-8c06-ade4d275b19a" (UID: "d39f7ee8-b620-4450-8c06-ade4d275b19a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:31.554383 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.554367    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:31.557430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.556520    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca" pod "d39f7ee8-b620-4450-8c06-ade4d275b19a" (UID: "d39f7ee8-b620-4450-8c06-ade4d275b19a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:31.557585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.557506    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl" pod "d39f7ee8-b620-4450-8c06-ade4d275b19a" (UID: "d39f7ee8-b620-4450-8c06-ade4d275b19a"). InnerVolumeSpecName "kube-api-access-vdfpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:31.655382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.655351    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d39f7ee8-b620-4450-8c06-ade4d275b19a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:31.655382 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.655379    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfpl\" (UniqueName: \"kubernetes.io/projected/d39f7ee8-b620-4450-8c06-ade4d275b19a-kube-api-access-vdfpl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:31.924221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.923428    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 31 01:54:31.924221 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:31.923488    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 31 01:54:32.148070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.148042    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"
Oct 31 01:54:32.148284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.148040    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch" event={"ID":"d39f7ee8-b620-4450-8c06-ade4d275b19a","Type":"ContainerDied","Data":"bf94a65b15d784f3bebe123a9d6457b10341a380a9d908bc4fac6e7a584195d9"}
Oct 31 01:54:32.148284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.148194    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf94a65b15d784f3bebe123a9d6457b10341a380a9d908bc4fac6e7a584195d9"
Oct 31 01:54:32.166415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.166388    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 31 01:54:32.166533 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.166440    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 31 01:54:32.925201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.924967    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 31 01:54:32.925201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:32.925035    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 31 01:54:33.166935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:33.166881    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 31 01:54:33.167157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:33.166952    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 31 01:54:33.919725 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:33.919692    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 31 01:54:33.919919 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:33.919746    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 31 01:54:34.166277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:34.166238    5419 patch_prober.go:29] interesting pod/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 31 01:54:34.166669 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:34.166300    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 31 01:54:35.206512 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:35.206470    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:35.207075 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:35.206988    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:35.208010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:35.207990    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 01:54:35.567044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:35.567011    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:35.568061 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:35.568036    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:35.569630 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:35.569609    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 01:54:37.254290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.254262    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"]
Oct 31 01:54:37.255286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.255255    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d39f7ee8-b620-4450-8c06-ade4d275b19a" containerName="revoke-keys"
Oct 31 01:54:37.255286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.255277    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39f7ee8-b620-4450-8c06-ade4d275b19a" containerName="revoke-keys"
Oct 31 01:54:37.255492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.255389    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39f7ee8-b620-4450-8c06-ade4d275b19a" containerName="revoke-keys"
Oct 31 01:54:37.259974 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.259950    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.314913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.314885    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"]
Oct 31 01:54:37.364890 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.364857    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-476ce2\""
Oct 31 01:54:37.367880 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.366520    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-476ce2-dockercfg-ct9pz\""
Oct 31 01:54:37.373198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.373155    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggdmd\" (UniqueName: \"kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd\") pod \"payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.373608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.373577    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.456217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.456182    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"]
Oct 31 01:54:37.476277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.476239    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.477620 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.477593    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggdmd\" (UniqueName: \"kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd\") pod \"payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.478174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.477365    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.486635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.486604    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.491324 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.491301    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"]
Oct 31 01:54:37.535028 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.534963    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggdmd\" (UniqueName: \"kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd\") pod \"payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.573816 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.573788    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:37.578973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.578954    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.579048 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.578998    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4g5\" (UniqueName: \"kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5\") pod \"payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.680341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.680303    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.680606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.680392    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4g5\" (UniqueName: \"kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5\") pod \"payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.682698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.682663    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.737360 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.737330    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4g5\" (UniqueName: \"kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5\") pod \"payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.811009 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.810917    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:37.958436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:37.958404    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"]
Oct 31 01:54:37.958892 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:37.958868    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68601002_b9af_4a6f_9968_33faf79bde28.slice/crio-c7d71fc06cf1399ff494989af9a58f72d37c192e6b2c8fe7b97d7412b262beb2 WatchSource:0}: Error finding container c7d71fc06cf1399ff494989af9a58f72d37c192e6b2c8fe7b97d7412b262beb2: Status 404 returned error can't find the container with id c7d71fc06cf1399ff494989af9a58f72d37c192e6b2c8fe7b97d7412b262beb2
Oct 31 01:54:38.185937 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:38.185914    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"]
Oct 31 01:54:38.186529 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:38.186504    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc76441f_c18a_4232_bf1d_e899ee0b4b3c.slice/crio-1534b4165e1053d449776598ab490b4748b015993be2e4145332a276eea69da4 WatchSource:0}: Error finding container 1534b4165e1053d449776598ab490b4748b015993be2e4145332a276eea69da4: Status 404 returned error can't find the container with id 1534b4165e1053d449776598ab490b4748b015993be2e4145332a276eea69da4
Oct 31 01:54:38.217343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:38.217318    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl" event={"ID":"fc76441f-c18a-4232-bf1d-e899ee0b4b3c","Type":"ContainerStarted","Data":"1534b4165e1053d449776598ab490b4748b015993be2e4145332a276eea69da4"}
Oct 31 01:54:38.218874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:38.218844    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf" event={"ID":"68601002-b9af-4a6f-9968-33faf79bde28","Type":"ContainerStarted","Data":"3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2"}
Oct 31 01:54:38.218965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:38.218884    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf" event={"ID":"68601002-b9af-4a6f-9968-33faf79bde28","Type":"ContainerStarted","Data":"c7d71fc06cf1399ff494989af9a58f72d37c192e6b2c8fe7b97d7412b262beb2"}
Oct 31 01:54:38.218965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:38.218913    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:38.336124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:38.336054    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf" podStartSLOduration=1.336041515 podStartE2EDuration="1.336041515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:38.332483503 +0000 UTC m=+2999.633440080" watchObservedRunningTime="2026-10-31 01:54:38.336041515 +0000 UTC m=+2999.636998091"
Oct 31 01:54:39.228572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:39.228538    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl" event={"ID":"fc76441f-c18a-4232-bf1d-e899ee0b4b3c","Type":"ContainerStarted","Data":"6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a"}
Oct 31 01:54:39.228792 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:39.228753    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:39.343012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:39.342955    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl" podStartSLOduration=2.34294137 podStartE2EDuration="2.34294137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:39.341095342 +0000 UTC m=+3000.642051932" watchObservedRunningTime="2026-10-31 01:54:39.34294137 +0000 UTC m=+3000.643897961"
Oct 31 01:54:42.170968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.170420    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"]
Oct 31 01:54:42.176935 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.176908    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:54:42.177079 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.177020    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.217154 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.217108    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"]
Oct 31 01:54:42.244828 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.244794    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-98b70c\""
Oct 31 01:54:42.282875 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.282834    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.308939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.308906    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njzm5\" (UniqueName: \"kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5\") pod \"payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.319495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.309334    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-98b70c-dockercfg-hq2dv\""
Oct 31 01:54:42.419984 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.419945    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.420246 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.420045    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njzm5\" (UniqueName: \"kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5\") pod \"payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.420739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.420717    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.477901 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.477863    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njzm5\" (UniqueName: \"kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5\") pod \"payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.491888 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.491858    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:42.870313 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:42.870285    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"]
Oct 31 01:54:42.871085 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:42.871050    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abbc023_4e14_45e3_a2ca_2ca0e4698d0d.slice/crio-4589e7b89e7d5eb51d53c75a957131bd2495f29f734035c77878af8ee93452e5 WatchSource:0}: Error finding container 4589e7b89e7d5eb51d53c75a957131bd2495f29f734035c77878af8ee93452e5: Status 404 returned error can't find the container with id 4589e7b89e7d5eb51d53c75a957131bd2495f29f734035c77878af8ee93452e5
Oct 31 01:54:43.392160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:43.391311    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq" event={"ID":"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d","Type":"ContainerStarted","Data":"05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df"}
Oct 31 01:54:43.392160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:43.391440    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq" event={"ID":"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d","Type":"ContainerStarted","Data":"4589e7b89e7d5eb51d53c75a957131bd2495f29f734035c77878af8ee93452e5"}
Oct 31 01:54:43.405164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:43.404370    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:43.556168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:43.544436    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq" podStartSLOduration=1.5444163 podStartE2EDuration="1.5444163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:43.53769535 +0000 UTC m=+3004.838651930" watchObservedRunningTime="2026-10-31 01:54:43.5444163 +0000 UTC m=+3004.845372879"
Oct 31 01:54:44.328946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.328911    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"]
Oct 31 01:54:44.339773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.339743    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482538    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482613    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6xc\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482648    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482725    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482762    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482791    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482873    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482912    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.483290 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.482938    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.590961    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591062    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591130    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6xc\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591188    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591278    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591320    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591352    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591448    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.591487    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.592084    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.619262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.592724    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.736272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.735534    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.736272 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.735833    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.737957 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.737628    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.767407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.767302    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"]
Oct 31 01:54:44.855726 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.839527    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.892218 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.890364    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:44.937164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:44.927362    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-051514-openshift-default-dockercfg-7trbv\""
Oct 31 01:54:45.005168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:45.000120    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:45.088165 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:54:45.086699    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:54:45.114164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:45.089037    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6xc\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc\") pod \"e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") " pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:45.198211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:45.192302    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:45.484281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:45.484234    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:54:46.148059 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:46.148027    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689fb428_fba4_418f_b221_a7498437f213.slice/crio-70b49efa106946f4376bbfe1e42344c9f69f52e9244c0913a1e23372831525db WatchSource:0}: Error finding container 70b49efa106946f4376bbfe1e42344c9f69f52e9244c0913a1e23372831525db: Status 404 returned error can't find the container with id 70b49efa106946f4376bbfe1e42344c9f69f52e9244c0913a1e23372831525db
Oct 31 01:54:46.148624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.148600    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"]
Oct 31 01:54:46.151294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.151257    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:46.151400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.151345    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:46.151400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.151385    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:54:46.571889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.571857    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" event={"ID":"689fb428-fba4-418f-b221-a7498437f213","Type":"ContainerStarted","Data":"b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"}
Oct 31 01:54:46.572391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.571909    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" event={"ID":"689fb428-fba4-418f-b221-a7498437f213","Type":"ContainerStarted","Data":"70b49efa106946f4376bbfe1e42344c9f69f52e9244c0913a1e23372831525db"}
Oct 31 01:54:46.700191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.700122    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podStartSLOduration=2.70010625 podStartE2EDuration="2.70010625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:46.687197169 +0000 UTC m=+3007.988153747" watchObservedRunningTime="2026-10-31 01:54:46.70010625 +0000 UTC m=+3008.001062877"
Oct 31 01:54:46.769000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.768966    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"]
Oct 31 01:54:46.773358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.773337    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:46.828217 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.827175    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:54:46.839409 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.839376    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:54:46.847333 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.839818    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:54:46.847579 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.839877    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:54:46.887672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.885981    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"]
Oct 31 01:54:46.922312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.922285    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:46.922504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.922319    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:46.922576 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:46.922566    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpxl\" (UniqueName: \"kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.023528 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.023497    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpxl\" (UniqueName: \"kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.023732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.023573    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.023732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.023609    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.024341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.024312    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.026371 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.026349    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.080434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.080373    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpxl\" (UniqueName: \"kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl\") pod \"maas-api-revoke-sub-3d9e3794-qcf7v\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.087256 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.087231    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:47.193038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.192998    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:47.198377 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.198348    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:47.465322 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:54:47.465292    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549a4e9b_0fef_4712_9941_ed5a2e51a0b9.slice/crio-a8bdcc4b365ab368b336d44a48e899b3ac28c8efc925e7094e6b1e3266dc52a4 WatchSource:0}: Error finding container a8bdcc4b365ab368b336d44a48e899b3ac28c8efc925e7094e6b1e3266dc52a4: Status 404 returned error can't find the container with id a8bdcc4b365ab368b336d44a48e899b3ac28c8efc925e7094e6b1e3266dc52a4
Oct 31 01:54:47.465453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.465330    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"]
Oct 31 01:54:47.581942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.581907    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v" event={"ID":"549a4e9b-0fef-4712-9941-ed5a2e51a0b9","Type":"ContainerStarted","Data":"a8bdcc4b365ab368b336d44a48e899b3ac28c8efc925e7094e6b1e3266dc52a4"}
Oct 31 01:54:47.582391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.582186    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:47.583226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:47.583206    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:54:48.593124 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:48.593089    5419 generic.go:356] "Generic (PLEG): container finished" podID="549a4e9b-0fef-4712-9941-ed5a2e51a0b9" containerID="800b6805946999c7d4ef4f161044ca8c366aa35a5482f62977aa11bb71f87ec7" exitCode=0
Oct 31 01:54:48.593522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:48.593172    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v" event={"ID":"549a4e9b-0fef-4712-9941-ed5a2e51a0b9","Type":"ContainerDied","Data":"800b6805946999c7d4ef4f161044ca8c366aa35a5482f62977aa11bb71f87ec7"}
Oct 31 01:54:49.232070 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.232042    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:54:49.753227 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.753202    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:49.858033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.857950    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl\" (UniqueName: \"kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl\") pod \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") "
Oct 31 01:54:49.858033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.857995    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca\") pod \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") "
Oct 31 01:54:49.858337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.858174    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca\") pod \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\" (UID: \"549a4e9b-0fef-4712-9941-ed5a2e51a0b9\") "
Oct 31 01:54:49.858495 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.858471    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca" pod "549a4e9b-0fef-4712-9941-ed5a2e51a0b9" (UID: "549a4e9b-0fef-4712-9941-ed5a2e51a0b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:54:49.858747 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.858729    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:49.860308 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.860287    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca" pod "549a4e9b-0fef-4712-9941-ed5a2e51a0b9" (UID: "549a4e9b-0fef-4712-9941-ed5a2e51a0b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:54:49.860414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.860327    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl" pod "549a4e9b-0fef-4712-9941-ed5a2e51a0b9" (UID: "549a4e9b-0fef-4712-9941-ed5a2e51a0b9"). InnerVolumeSpecName "kube-api-access-6mpxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:54:49.960031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.960003    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:49.960031 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:49.960029    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mpxl\" (UniqueName: \"kubernetes.io/projected/549a4e9b-0fef-4712-9941-ed5a2e51a0b9-kube-api-access-6mpxl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:54:50.238729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:50.238694    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:54:50.609748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:50.609660    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"
Oct 31 01:54:50.609748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:50.609654    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v" event={"ID":"549a4e9b-0fef-4712-9941-ed5a2e51a0b9","Type":"ContainerDied","Data":"a8bdcc4b365ab368b336d44a48e899b3ac28c8efc925e7094e6b1e3266dc52a4"}
Oct 31 01:54:50.609748 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:50.609716    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8bdcc4b365ab368b336d44a48e899b3ac28c8efc925e7094e6b1e3266dc52a4"
Oct 31 01:54:54.918301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:54.918274    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 01:54:57.007966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:57.007923    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"]
Oct 31 01:54:57.057426 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:57.057387    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-c0daf4-5k5c9"]
Oct 31 01:54:58.838442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:54:58.838401    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1992f5-1016-4a04-a9eb-f523ca855ab4" path="/var/lib/kubelet/pods/9c1992f5-1016-4a04-a9eb-f523ca855ab4/volumes"
Oct 31 01:55:05.288657 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.288607    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"]
Oct 31 01:55:05.290157 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.290116    5419 cpu_manager.go:414] "containerMap: removing container" podUID="549a4e9b-0fef-4712-9941-ed5a2e51a0b9" containerName="revoke-keys"
Oct 31 01:55:05.290319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.290170    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549a4e9b-0fef-4712-9941-ed5a2e51a0b9" containerName="revoke-keys"
Oct 31 01:55:05.290582 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.290565    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="549a4e9b-0fef-4712-9941-ed5a2e51a0b9" containerName="revoke-keys"
Oct 31 01:55:05.297314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.297291    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.380611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.380580    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"]
Oct 31 01:55:05.397852 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.397816    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.397968    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398077    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398182 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398129    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398213    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrmd\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398374 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398362    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398391    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398413    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.398487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.398455    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499488    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499525    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499544    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499576    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499626    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499711    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499765    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499791    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.499823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.499813    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrmd\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.500236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.500048    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.500236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.500103    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.500236 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.500195    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.500575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.500369    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.500575 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.500457    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.502229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.502206    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.502430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.502412    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.556672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.556594    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.556855 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.556823    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrmd\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd\") pod \"e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") " pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:05.612462 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:05.612427    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:06.049854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.049820    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"]
Oct 31 01:55:06.050114 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:55:06.050093    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbc5c62_f0c9_4831_8fe3_4e429de2a943.slice/crio-ffcc153bda14436fc01e22dd797295241719af369bef77244bb21cfc40aa54d4 WatchSource:0}: Error finding container ffcc153bda14436fc01e22dd797295241719af369bef77244bb21cfc40aa54d4: Status 404 returned error can't find the container with id ffcc153bda14436fc01e22dd797295241719af369bef77244bb21cfc40aa54d4
Oct 31 01:55:06.052623 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.052590    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:55:06.052728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.052653    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:55:06.052728 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.052703    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:55:06.731111 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.731071    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" event={"ID":"7fbc5c62-f0c9-4831-8fe3-4e429de2a943","Type":"ContainerStarted","Data":"82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd"}
Oct 31 01:55:06.731605 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.731128    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" event={"ID":"7fbc5c62-f0c9-4831-8fe3-4e429de2a943","Type":"ContainerStarted","Data":"ffcc153bda14436fc01e22dd797295241719af369bef77244bb21cfc40aa54d4"}
Oct 31 01:55:06.960821 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:06.960770    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podStartSLOduration=1.960754414 podStartE2EDuration="1.960754414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:06.910283215 +0000 UTC m=+3028.211239792" watchObservedRunningTime="2026-10-31 01:55:06.960754414 +0000 UTC m=+3028.261710990"
Oct 31 01:55:07.613304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:07.613272    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:07.615041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:07.615011    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 31 01:55:07.615211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:07.615072    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 31 01:55:08.612715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:08.612681    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 31 01:55:08.613184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:08.612733    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 31 01:55:09.615159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:09.614961    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 31 01:55:09.615159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:09.615036    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 31 01:55:10.615632 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:10.615581    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 31 01:55:10.616304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:10.616271    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 31 01:55:11.619634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:11.619581    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 31 01:55:11.620316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:11.620282    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 31 01:55:12.379066 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:12.379032    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"]
Oct 31 01:55:12.613497 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:12.613467    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Oct 31 01:55:12.613662 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:12.613518    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Oct 31 01:55:13.617501 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:13.617465    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:13.617981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:13.617929    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:13.618550 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:13.618529    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 01:55:14.583315 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:55:14.583280    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:55:18.635520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.635488    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"]
Oct 31 01:55:18.641265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.641227    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.697030 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.696994    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9e3b95-openshift-default-dockercfg-7zdzl\""
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.774986    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775043    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775085    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775114    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775221    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775243    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775381    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltq9\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775448    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.855194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.775475    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877370    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltq9\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877449    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877477    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877578    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877613    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877650    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877676    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877767    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.877790    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.878301    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.878651    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.878723    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.879177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.878884    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.880560 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.880535    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.961750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.961673    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"]
Oct 31 01:55:18.968000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.967929    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.968956 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.968581    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.981339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.981303    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:18.983641 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:18.983569    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltq9\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9\") pod \"e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") " pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:19.256362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.256333    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:19.693194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.693166    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"]
Oct 31 01:55:19.693692 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:55:19.693668    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dc283f_bbcb_4544_8f81_13a0d10c52f6.slice/crio-bac81672f321151aa9805b5b7b330cb6bb6abce82a19fce0a3d4c3b037c86357 WatchSource:0}: Error finding container bac81672f321151aa9805b5b7b330cb6bb6abce82a19fce0a3d4c3b037c86357: Status 404 returned error can't find the container with id bac81672f321151aa9805b5b7b330cb6bb6abce82a19fce0a3d4c3b037c86357
Oct 31 01:55:19.696185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.696131    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:55:19.696273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.696224    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:55:19.696273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.696262    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:55:19.889180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.889116    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" event={"ID":"38dc283f-bbcb-4544-8f81-13a0d10c52f6","Type":"ContainerStarted","Data":"6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"}
Oct 31 01:55:19.889370 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:19.889204    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" event={"ID":"38dc283f-bbcb-4544-8f81-13a0d10c52f6","Type":"ContainerStarted","Data":"bac81672f321151aa9805b5b7b330cb6bb6abce82a19fce0a3d4c3b037c86357"}
Oct 31 01:55:20.009822 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:20.009774    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podStartSLOduration=2.009760003 podStartE2EDuration="2.009760003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:20.004932267 +0000 UTC m=+3041.305888843" watchObservedRunningTime="2026-10-31 01:55:20.009760003 +0000 UTC m=+3041.310716580"
Oct 31 01:55:20.256777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:20.256740    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:20.262776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:20.262711    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:20.898024 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:20.897995    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:20.899045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:20.899027    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:55:21.143896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:21.143861    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"]
Oct 31 01:55:21.144117 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:21.144080    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy" containerID="cri-o://62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b" gracePeriod=30
Oct 31 01:55:26.468100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.468072    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:55:26.607659 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607631    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.607874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607703    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.607874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607743    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.607874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607785    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.607874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607844    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.608129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607889    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.608129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.607987    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.608129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608026    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.608129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608035    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:26.608129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608052    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:26.608129 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608055    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket\") pod \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\" (UID: \"51dc475d-a4c7-4510-9ebd-98d03c4adbe9\") "
Oct 31 01:55:26.608672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608292    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:26.608672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608473    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:26.608672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.608497    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:55:26.609102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.609077    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.609234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.609108    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.609234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.609123    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.609234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.609155    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.609234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.609170    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.610300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.610271    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:26.610446 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.610428    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:26.610776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.610750    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:55:26.610776 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.610768    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t" pod "51dc475d-a4c7-4510-9ebd-98d03c4adbe9" (UID: "51dc475d-a4c7-4510-9ebd-98d03c4adbe9"). InnerVolumeSpecName "kube-api-access-svv7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:26.710660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.710585    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.710660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.710613    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.710660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.710623    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.710660 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.710632    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svv7t\" (UniqueName: \"kubernetes.io/projected/51dc475d-a4c7-4510-9ebd-98d03c4adbe9-kube-api-access-svv7t\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:26.948375 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.948341    5419 generic.go:356] "Generic (PLEG): container finished" podID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerID="62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b" exitCode=0
Oct 31 01:55:26.948573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.948398    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" event={"ID":"51dc475d-a4c7-4510-9ebd-98d03c4adbe9","Type":"ContainerDied","Data":"62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b"}
Oct 31 01:55:26.948573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.948443    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"
Oct 31 01:55:26.948573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.948462    5419 scope.go:122] "RemoveContainer" containerID="62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b"
Oct 31 01:55:26.948573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.948448    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9" event={"ID":"51dc475d-a4c7-4510-9ebd-98d03c4adbe9","Type":"ContainerDied","Data":"8506f81162e5158dfa2576812cfd6fadef9436975412bbe7c145dca91d81fc80"}
Oct 31 01:55:26.959235 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.959216    5419 scope.go:122] "RemoveContainer" containerID="afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"
Oct 31 01:55:26.968788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.968771    5419 scope.go:122] "RemoveContainer" containerID="62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b"
Oct 31 01:55:26.969053 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:55:26.969034    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b\": container with ID starting with 62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b not found: ID does not exist" containerID="62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b"
Oct 31 01:55:26.969104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.969062    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b"} err="failed to get container status \"62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b\": rpc error: code = NotFound desc = could not find container \"62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b\": container with ID starting with 62ebcb3650d1f242b56139e0d6f0b0ed0429f7bc0f1b51606f47b4eebe061e1b not found: ID does not exist"
Oct 31 01:55:26.969104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.969080    5419 scope.go:122] "RemoveContainer" containerID="afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"
Oct 31 01:55:26.969356 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:55:26.969339    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30\": container with ID starting with afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30 not found: ID does not exist" containerID="afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"
Oct 31 01:55:26.969395 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:26.969363    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30"} err="failed to get container status \"afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30\": rpc error: code = NotFound desc = could not find container \"afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30\": container with ID starting with afe4e25b974ff9fd3f7d590c283d84b4fe8c7e4169e6e0e718875c9d93c1ff30 not found: ID does not exist"
Oct 31 01:55:27.114552 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.114523    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"]
Oct 31 01:55:27.163251 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.163224    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dbhh9"]
Oct 31 01:55:27.400788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.400751    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" podUID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" containerName="main" containerID="cri-o://17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940" gracePeriod=45
Oct 31 01:55:27.680757 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.680726    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:55:27.822870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.822834    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r\" (UniqueName: \"kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r\") pod \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") "
Oct 31 01:55:27.823068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.822974    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home\") pod \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") "
Oct 31 01:55:27.823068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823018    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm\") pod \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") "
Oct 31 01:55:27.823216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823126    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs\" (UniqueName: \"kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs\") pod \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") "
Oct 31 01:55:27.823216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823196    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache\") pod \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") "
Oct 31 01:55:27.823332 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823276    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home" pod "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" (UID: "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:27.823398 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823347    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir\") pod \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\" (UID: \"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3\") "
Oct 31 01:55:27.824036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823724    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir" pod "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" (UID: "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:27.824036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823788    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache" pod "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" (UID: "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:27.824036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823962    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.824036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.823974    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.824036 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.824009    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.826042 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.826012    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm" pod "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" (UID: "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:55:27.826202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.826042    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r" pod "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" (UID: "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3"). InnerVolumeSpecName "kube-api-access-hw78r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:55:27.826202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.826078    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs" pod "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" (UID: "bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:55:27.925530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.925260    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.925530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.925291    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.925530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.925305    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw78r\" (UniqueName: \"kubernetes.io/projected/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3-kube-api-access-hw78r\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:55:27.956848 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.956818    5419 generic.go:356] "Generic (PLEG): container finished" podID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" containerID="17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940" exitCode=0
Oct 31 01:55:27.957040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.956862    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"
Oct 31 01:55:27.957040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.956899    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" event={"ID":"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3","Type":"ContainerDied","Data":"17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940"}
Oct 31 01:55:27.957040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.956941    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" event={"ID":"bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3","Type":"ContainerDied","Data":"f9eb1ce820e2213d98f6f4ebb5db1e0e45108f83c7c67ab04aee42d091548892"}
Oct 31 01:55:27.957040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.956964    5419 scope.go:122] "RemoveContainer" containerID="17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940"
Oct 31 01:55:27.969229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.969198    5419 scope.go:122] "RemoveContainer" containerID="17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940"
Oct 31 01:55:27.969494 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:55:27.969475    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940\": container with ID starting with 17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940 not found: ID does not exist" containerID="17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940"
Oct 31 01:55:27.969561 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:27.969504    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940"} err="failed to get container status \"17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940\": rpc error: code = NotFound desc = could not find container \"17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940\": container with ID starting with 17347f59aeac9a0115e31fd99901b44d001c2ab585b7c8ebfd0c0926f0d8e940 not found: ID does not exist"
Oct 31 01:55:28.120040 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:28.120003    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"]
Oct 31 01:55:28.123505 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:28.123485    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2"]
Oct 31 01:55:28.500114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:28.500081    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:55:28.581551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:28.581513    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-c0daf4/praxis-route-c0daf4-kserve-69c46b7545-2mqw2" podUID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.221:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:55:28.839003 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:28.838933    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" path="/var/lib/kubelet/pods/51dc475d-a4c7-4510-9ebd-98d03c4adbe9/volumes"
Oct 31 01:55:28.839548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:28.839531    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" path="/var/lib/kubelet/pods/bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3/volumes"
Oct 31 01:55:36.103279 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.103246    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"]
Oct 31 01:55:36.103925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.103910    5419 cpu_manager.go:414] "containerMap: removing container" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy"
Oct 31 01:55:36.103986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.103927    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy"
Oct 31 01:55:36.103986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.103936    5419 cpu_manager.go:414] "containerMap: removing container" podUID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" containerName="main"
Oct 31 01:55:36.103986 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.103941    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" containerName="main"
Oct 31 01:55:36.104100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.104067    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy"
Oct 31 01:55:36.104100 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.104079    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce7ceea-4c5a-4cf7-8d69-887ce9c92ed3" containerName="main"
Oct 31 01:55:36.109126 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.109107    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.178392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.175394    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"]
Oct 31 01:55:36.217194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.212385    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9e3b95\""
Oct 31 01:55:36.217194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.213165    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9e3b95-dockercfg-59jkt\""
Oct 31 01:55:36.308163 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.285686    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjlth\" (UniqueName: \"kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth\") pod \"payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.308671 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.308639    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.333520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.333479    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"]
Oct 31 01:55:36.348618 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.348567    5419 cpu_manager.go:414] "containerMap: removing container" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy"
Oct 31 01:55:36.348922 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.348899    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy"
Oct 31 01:55:36.349474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.349438    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="51dc475d-a4c7-4510-9ebd-98d03c4adbe9" containerName="istio-proxy"
Oct 31 01:55:36.367583 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.367494    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.383351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.383030    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"]
Oct 31 01:55:36.434959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.434923    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.435184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.435126    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjlth\" (UniqueName: \"kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth\") pod \"payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.435715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.435690    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.510396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.510364    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjlth\" (UniqueName: \"kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth\") pod \"payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:36.536682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.536647    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhd5\" (UniqueName: \"kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5\") pod \"payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.536906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.536888    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.637709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.637625    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.637709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.637685    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhd5\" (UniqueName: \"kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5\") pod \"payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.638402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.638382    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.694253 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.694221    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhd5\" (UniqueName: \"kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5\") pod \"payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.698084 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.698057    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:36.722732 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:36.722698    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:37.118425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:37.118395    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"]
Oct 31 01:55:37.118808 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:55:37.118745    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b80986f_569d_4cf1_b128_b01992b59b59.slice/crio-857656d7f00c84d6e7466f5f0ae651b4a295dd453db53096c913c0e80772efdd WatchSource:0}: Error finding container 857656d7f00c84d6e7466f5f0ae651b4a295dd453db53096c913c0e80772efdd: Status 404 returned error can't find the container with id 857656d7f00c84d6e7466f5f0ae651b4a295dd453db53096c913c0e80772efdd
Oct 31 01:55:37.187613 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:37.187583    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"]
Oct 31 01:55:37.190371 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:55:37.190345    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478fa64f_fb3c_42ed_b828_e084c661362a.slice/crio-ad55ccddd4f5e933d2ffb81c4f351835c38660c8cb5f0dbcf7c977c5aacb3a38 WatchSource:0}: Error finding container ad55ccddd4f5e933d2ffb81c4f351835c38660c8cb5f0dbcf7c977c5aacb3a38: Status 404 returned error can't find the container with id ad55ccddd4f5e933d2ffb81c4f351835c38660c8cb5f0dbcf7c977c5aacb3a38
Oct 31 01:55:38.079150 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.079096    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq" event={"ID":"0b80986f-569d-4cf1-b128-b01992b59b59","Type":"ContainerStarted","Data":"e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f"}
Oct 31 01:55:38.079345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.079187    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:38.079345 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.079213    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq" event={"ID":"0b80986f-569d-4cf1-b128-b01992b59b59","Type":"ContainerStarted","Data":"857656d7f00c84d6e7466f5f0ae651b4a295dd453db53096c913c0e80772efdd"}
Oct 31 01:55:38.080933 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.080901    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4" event={"ID":"478fa64f-fb3c-42ed-b828-e084c661362a","Type":"ContainerStarted","Data":"9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d"}
Oct 31 01:55:38.081063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.080941    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4" event={"ID":"478fa64f-fb3c-42ed-b828-e084c661362a","Type":"ContainerStarted","Data":"ad55ccddd4f5e933d2ffb81c4f351835c38660c8cb5f0dbcf7c977c5aacb3a38"}
Oct 31 01:55:38.081063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.080967    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:38.192799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.192732    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq" podStartSLOduration=2.192557243 podStartE2EDuration="2.192557243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:38.191769906 +0000 UTC m=+3059.492726483" watchObservedRunningTime="2026-10-31 01:55:38.192557243 +0000 UTC m=+3059.493513822"
Oct 31 01:55:38.327556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:38.327472    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4" podStartSLOduration=2.327450159 podStartE2EDuration="2.327450159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:38.3199157 +0000 UTC m=+3059.620872282" watchObservedRunningTime="2026-10-31 01:55:38.327450159 +0000 UTC m=+3059.628406738"
Oct 31 01:55:40.993475 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:40.993440    5419 scope.go:122] "RemoveContainer" containerID="49c61d798e78fc6226fd7dd62e6414c911af1f6e6fb28b96147de1895a8dfe1f"
Oct 31 01:55:41.004190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:41.004170    5419 scope.go:122] "RemoveContainer" containerID="b86259845834a2facbd7dfee2b1da5fc58f27b3c35c6ccdbe50a6cdbe720d7c1"
Oct 31 01:55:44.582622 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:55:44.582586    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:55:49.091234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:49.091159    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 01:55:49.091234 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:49.091226    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 01:55:55.363999 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:55.363965    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"]
Oct 31 01:55:55.364415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:55:55.364235    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy" containerID="cri-o://90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce" gracePeriod=30
Oct 31 01:56:00.634402 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.634346    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:56:00.748652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748621    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.748838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748689    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.748838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748716    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.748838 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748804    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.749019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748870    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.749019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748892    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.749019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.748965    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.749019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749003    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749094    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749124    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749180    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert\") pod \"05db315c-10b9-4a39-8656-64ed2a590be1\" (UID: \"05db315c-10b9-4a39-8656-64ed2a590be1\") "
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749275    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749362    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749593    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749768    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749782    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749792    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749801    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05db315c-10b9-4a39-8656-64ed2a590be1-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.749916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.749809    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.751255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.751233    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:56:00.751428 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.751406    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:00.751789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.751770    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:56:00.751789 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.751774    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr" pod "05db315c-10b9-4a39-8656-64ed2a590be1" (UID: "05db315c-10b9-4a39-8656-64ed2a590be1"). InnerVolumeSpecName "kube-api-access-nncbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:00.851743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.851653    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.851743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.851684    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05db315c-10b9-4a39-8656-64ed2a590be1-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.851743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.851695    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05db315c-10b9-4a39-8656-64ed2a590be1-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:00.851743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:00.851711    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nncbr\" (UniqueName: \"kubernetes.io/projected/05db315c-10b9-4a39-8656-64ed2a590be1-kube-api-access-nncbr\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:01.256049 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.256016    5419 generic.go:356] "Generic (PLEG): container finished" podID="05db315c-10b9-4a39-8656-64ed2a590be1" containerID="90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce" exitCode=0
Oct 31 01:56:01.256273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.256067    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"
Oct 31 01:56:01.256273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.256104    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" event={"ID":"05db315c-10b9-4a39-8656-64ed2a590be1","Type":"ContainerDied","Data":"90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce"}
Oct 31 01:56:01.256273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.256176    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55" event={"ID":"05db315c-10b9-4a39-8656-64ed2a590be1","Type":"ContainerDied","Data":"62594ecb779f211079f2931ead0a36ef7deb88a50d5014cca3833b96253b1d80"}
Oct 31 01:56:01.256273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.256208    5419 scope.go:122] "RemoveContainer" containerID="90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce"
Oct 31 01:56:01.267867 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.267848    5419 scope.go:122] "RemoveContainer" containerID="e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"
Oct 31 01:56:01.277716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.277697    5419 scope.go:122] "RemoveContainer" containerID="90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce"
Oct 31 01:56:01.278006 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:56:01.277987    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce\": container with ID starting with 90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce not found: ID does not exist" containerID="90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce"
Oct 31 01:56:01.278089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.278020    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce"} err="failed to get container status \"90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce\": rpc error: code = NotFound desc = could not find container \"90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce\": container with ID starting with 90728c5393e9cd2f4d67b1af10104a719e30ac88e44c3d51e0f26320e31a95ce not found: ID does not exist"
Oct 31 01:56:01.278089 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.278046    5419 scope.go:122] "RemoveContainer" containerID="e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"
Oct 31 01:56:01.278339 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:56:01.278320    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2\": container with ID starting with e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2 not found: ID does not exist" containerID="e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"
Oct 31 01:56:01.278396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.278345    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2"} err="failed to get container status \"e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2\": rpc error: code = NotFound desc = could not find container \"e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2\": container with ID starting with e97d4db630e8907438ebe22c723ff5132fc69a8c0f395f196c66658472ea2fc2 not found: ID does not exist"
Oct 31 01:56:01.423573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.423544    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"]
Oct 31 01:56:01.473392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:01.473357    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c0daf4-openshift-default-84d68dc6f-pkq55"]
Oct 31 01:56:02.838362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:02.838330    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" path="/var/lib/kubelet/pods/05db315c-10b9-4a39-8656-64ed2a590be1/volumes"
Oct 31 01:56:10.890904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.890871    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"]
Oct 31 01:56:10.891477 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.891460    5419 cpu_manager.go:414] "containerMap: removing container" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy"
Oct 31 01:56:10.891524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.891478    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy"
Oct 31 01:56:10.891524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.891519    5419 cpu_manager.go:414] "containerMap: removing container" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy"
Oct 31 01:56:10.891587 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.891525    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy"
Oct 31 01:56:10.891635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.891631    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy"
Oct 31 01:56:10.891673 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.891643    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="05db315c-10b9-4a39-8656-64ed2a590be1" containerName="istio-proxy"
Oct 31 01:56:10.895062 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.895047    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:10.947321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.947290    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:56:10.947321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.947309    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:56:10.947538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.947486    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:56:10.947538 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.947497    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:56:10.952319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.952295    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"]
Oct 31 01:56:10.978421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.978387    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwsn5\" (UniqueName: \"kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:10.978640 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.978571    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:10.978700 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:10.978681    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.079492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.079459    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.079703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.079538    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsn5\" (UniqueName: \"kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.079703 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.079602    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.080244 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.080223    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.081911 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.081888    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.136056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.136029    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsn5\" (UniqueName: \"kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5\") pod \"maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.209474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.209393    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:11.587745 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:11.587712    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"]
Oct 31 01:56:11.588757 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:56:11.588733    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99f3ecc_bbe5_4ad2_aeaa_f5f04686eba8.slice/crio-f7f85b9d7148abcef87ecd752d3b4ac7c6b5b2df5849a75e598d8ea5a5cb9058 WatchSource:0}: Error finding container f7f85b9d7148abcef87ecd752d3b4ac7c6b5b2df5849a75e598d8ea5a5cb9058: Status 404 returned error can't find the container with id f7f85b9d7148abcef87ecd752d3b4ac7c6b5b2df5849a75e598d8ea5a5cb9058
Oct 31 01:56:12.354021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:12.353985    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2" event={"ID":"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8","Type":"ContainerStarted","Data":"7b0a22581ac9012d574a50803dc998297960bb5b0e21e33a1e3e124cda485a04"}
Oct 31 01:56:12.354440 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:12.354069    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2" event={"ID":"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8","Type":"ContainerStarted","Data":"f7f85b9d7148abcef87ecd752d3b4ac7c6b5b2df5849a75e598d8ea5a5cb9058"}
Oct 31 01:56:12.471179 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:12.469357    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2" podStartSLOduration=2.469337711 podStartE2EDuration="2.469337711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:12.466711497 +0000 UTC m=+3093.767668094" watchObservedRunningTime="2026-10-31 01:56:12.469337711 +0000 UTC m=+3093.770294288"
Oct 31 01:56:13.363486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:13.363454    5419 generic.go:356] "Generic (PLEG): container finished" podID="e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" containerID="7b0a22581ac9012d574a50803dc998297960bb5b0e21e33a1e3e124cda485a04" exitCode=0
Oct 31 01:56:13.363906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:13.363517    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2" event={"ID":"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8","Type":"ContainerDied","Data":"7b0a22581ac9012d574a50803dc998297960bb5b0e21e33a1e3e124cda485a04"}
Oct 31 01:56:14.522443 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.522414    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:14.582329 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:56:14.582295    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:56:14.622000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.621922    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca\") pod \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") "
Oct 31 01:56:14.622200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.622020    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5\" (UniqueName: \"kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5\") pod \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") "
Oct 31 01:56:14.622289 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.622222    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca\") pod \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\" (UID: \"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8\") "
Oct 31 01:56:14.622386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.622363    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca" pod "e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" (UID: "e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:14.622652 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.622634    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:14.624337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.624313    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca" pod "e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" (UID: "e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:56:14.624431 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.624360    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5" pod "e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" (UID: "e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8"). InnerVolumeSpecName "kube-api-access-qwsn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:14.723530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.723493    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:14.723530 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:14.723525    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwsn5\" (UniqueName: \"kubernetes.io/projected/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8-kube-api-access-qwsn5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:15.379229 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:15.379193    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"
Oct 31 01:56:15.379436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:15.379228    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2" event={"ID":"e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8","Type":"ContainerDied","Data":"f7f85b9d7148abcef87ecd752d3b4ac7c6b5b2df5849a75e598d8ea5a5cb9058"}
Oct 31 01:56:15.379436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:15.379276    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7f85b9d7148abcef87ecd752d3b4ac7c6b5b2df5849a75e598d8ea5a5cb9058"
Oct 31 01:56:16.501238 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:16.501194    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:56:34.393609 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:34.393562    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" podUID="7360d5bb713e28adcb32faa1b6003ade" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 31 01:56:41.049045 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:41.049008    5419 scope.go:122] "RemoveContainer" containerID="5079b29c3c09676020a34794579ba453cde1231c2729c5b948d146315459a22c"
Oct 31 01:56:42.619328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.619297    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"]
Oct 31 01:56:42.619903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.619889    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" containerName="revoke-keys"
Oct 31 01:56:42.619966 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.619906    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" containerName="revoke-keys"
Oct 31 01:56:42.620041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.620032    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" containerName="revoke-keys"
Oct 31 01:56:42.624925 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.624908    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.681386 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.681351    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"]
Oct 31 01:56:42.681724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.681701    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:56:42.725439 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.725406    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:56:42.725681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.725660    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:56:42.725783 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.725664    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:56:42.752764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.752729    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86zl\" (UniqueName: \"kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.752942 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.752785    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.753027 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.753000    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.854116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.854076    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86zl\" (UniqueName: \"kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.854329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.854133    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.854329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.854262    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.855020 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.854996    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.856715 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.856681    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.910164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.910080    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86zl\" (UniqueName: \"kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl\") pod \"maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:42.939702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:42.939667    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:43.323102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:43.323074    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"]
Oct 31 01:56:43.324562 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:56:43.324537    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb784c2c_0fe4_4c40_8fe4_e9a5d5922001.slice/crio-d60292cc61c49e76c08d56f9b4f4e204638f0ba89754d75769931dce39ddff9c WatchSource:0}: Error finding container d60292cc61c49e76c08d56f9b4f4e204638f0ba89754d75769931dce39ddff9c: Status 404 returned error can't find the container with id d60292cc61c49e76c08d56f9b4f4e204638f0ba89754d75769931dce39ddff9c
Oct 31 01:56:43.621083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:43.620999    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6" event={"ID":"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001","Type":"ContainerStarted","Data":"d60292cc61c49e76c08d56f9b4f4e204638f0ba89754d75769931dce39ddff9c"}
Oct 31 01:56:44.583338 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:56:44.583256    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:56:44.629754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:44.629723    5419 generic.go:356] "Generic (PLEG): container finished" podID="bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" containerID="7a902dc1d269bdadfabd84376e7c239f3f43b77b09d45879a426a9cf87dab0de" exitCode=0
Oct 31 01:56:44.630164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:44.629775    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6" event={"ID":"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001","Type":"ContainerDied","Data":"7a902dc1d269bdadfabd84376e7c239f3f43b77b09d45879a426a9cf87dab0de"}
Oct 31 01:56:45.795588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.795562    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:45.891837 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.891767    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca\") pod \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") "
Oct 31 01:56:45.891994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.891900    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl\" (UniqueName: \"kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl\") pod \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") "
Oct 31 01:56:45.891994 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.891928    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca\") pod \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\" (UID: \"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001\") "
Oct 31 01:56:45.892122 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.892099    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca" pod "bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" (UID: "bb784c2c-0fe4-4c40-8fe4-e9a5d5922001"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:56:45.892421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.892399    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:45.894223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.894196    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca" pod "bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" (UID: "bb784c2c-0fe4-4c40-8fe4-e9a5d5922001"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:56:45.894319 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.894292    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl" pod "bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" (UID: "bb784c2c-0fe4-4c40-8fe4-e9a5d5922001"). InnerVolumeSpecName "kube-api-access-w86zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:56:45.993830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.993797    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86zl\" (UniqueName: \"kubernetes.io/projected/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-kube-api-access-w86zl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:45.993830 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:45.993827    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:56:46.645685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:46.645646    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6" event={"ID":"bb784c2c-0fe4-4c40-8fe4-e9a5d5922001","Type":"ContainerDied","Data":"d60292cc61c49e76c08d56f9b4f4e204638f0ba89754d75769931dce39ddff9c"}
Oct 31 01:56:46.645870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:46.645693    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"
Oct 31 01:56:46.645870 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:46.645699    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60292cc61c49e76c08d56f9b4f4e204638f0ba89754d75769931dce39ddff9c"
Oct 31 01:56:57.909946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.909900    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"]
Oct 31 01:56:57.910806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.910783    5419 cpu_manager.go:414] "containerMap: removing container" podUID="bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" containerName="revoke-keys"
Oct 31 01:56:57.910926 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.910807    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" containerName="revoke-keys"
Oct 31 01:56:57.910987 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.910962    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" containerName="revoke-keys"
Oct 31 01:56:57.914701 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.914676    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.972556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.972518    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-98c748-openshift-default-dockercfg-9fw2c\""
Oct 31 01:56:57.986906 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.986871    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.987102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.986921    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.987102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987030    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.987102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987091    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.988277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987175    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.988277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987215    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.988277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987358    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.988277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987439    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dmz\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:57.988277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:57.987473    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.025835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.025803    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"]
Oct 31 01:56:58.108990 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.108943    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.109351 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.109327    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dmz\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.109499 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.109483    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.109637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.109621    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.109753 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.109738    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.109885 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.109869    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.110015 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.110000    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.110198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.110181    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.110339 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.110323    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.111965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.111940    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.117764 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.117738    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.124881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.124845    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.125168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.125123    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.125434 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.125410    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.132118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.132079    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.139403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.139375    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.173997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.173940    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dmz\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.180384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.180363    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token\") pod \"e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") " pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.236611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.236514    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:56:58.680494 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.680466    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"]
Oct 31 01:56:58.680787 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:56:58.680760    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f89ea9_39c8_4263_92e4_34417a61dc0a.slice/crio-b2acfa9cb2f6603fb522d1ff3a5db1b5b1d559896f5e81b0a6f02df8042a104a WatchSource:0}: Error finding container b2acfa9cb2f6603fb522d1ff3a5db1b5b1d559896f5e81b0a6f02df8042a104a: Status 404 returned error can't find the container with id b2acfa9cb2f6603fb522d1ff3a5db1b5b1d559896f5e81b0a6f02df8042a104a
Oct 31 01:56:58.683415 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.683381    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:56:58.683522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.683459    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:56:58.683522 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.683502    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:56:58.735033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:58.735006    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" event={"ID":"c8f89ea9-39c8-4263-92e4-34417a61dc0a","Type":"ContainerStarted","Data":"b2acfa9cb2f6603fb522d1ff3a5db1b5b1d559896f5e81b0a6f02df8042a104a"}
Oct 31 01:56:59.743881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:59.743851    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" event={"ID":"c8f89ea9-39c8-4263-92e4-34417a61dc0a","Type":"ContainerStarted","Data":"c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"}
Oct 31 01:56:59.909323 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:56:59.909272    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podStartSLOduration=2.909253671 podStartE2EDuration="2.909253671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:59.909179688 +0000 UTC m=+3141.210136266" watchObservedRunningTime="2026-10-31 01:56:59.909253671 +0000 UTC m=+3141.210210251"
Oct 31 01:57:00.238958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:00.238924    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:57:00.354503 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:00.354472    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:57:00.539404 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:00.539320    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:57:00.752645 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:00.752615    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:57:00.753635 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:00.753616    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:57:02.053746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:02.053700    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"]
Oct 31 01:57:02.103733 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:02.103693    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a62a293-bfq84"]
Oct 31 01:57:02.837874 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:02.837844    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7543ab5e-eaa7-4ade-ad23-c0e387518f3b" path="/var/lib/kubelet/pods/7543ab5e-eaa7-4ade-ad23-c0e387518f3b/volumes"
Oct 31 01:57:06.456327 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.456061    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"]
Oct 31 01:57:06.464650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.464625    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.505677 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.505642    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"]
Oct 31 01:57:06.529691 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.529664    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-98c748\""
Oct 31 01:57:06.626161 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.625190    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"]
Oct 31 01:57:06.670165 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.669457    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:06.684973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.682431    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhq2n\" (UniqueName: \"kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n\") pod \"payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.684973 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.682615    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.736107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.722480    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-98c748-dockercfg-xjjvw\""
Oct 31 01:57:06.744775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.744738    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"]
Oct 31 01:57:06.784607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.784559    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62gz8\" (UniqueName: \"kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8\") pod \"payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:06.784795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.784623    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.784847 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.784805    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhq2n\" (UniqueName: \"kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n\") pod \"payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.784903 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.784868    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:06.785459 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.785438    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.841094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.841061    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhq2n\" (UniqueName: \"kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n\") pod \"payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:06.885895 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.885862    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62gz8\" (UniqueName: \"kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8\") pod \"payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:06.886073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.886025    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:06.886731 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.886710    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:06.945294 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:06.945268    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62gz8\" (UniqueName: \"kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8\") pod \"payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:07.058478 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.058449    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:07.103508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.103478    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:07.484301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.484269    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"]
Oct 31 01:57:07.546204 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.546180    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"]
Oct 31 01:57:07.547484 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:07.547459    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6558d2_2bfa_4ace_af46_60e0c3b41415.slice/crio-f8e0f64348b2f0f7258fc78e1a688b3fa9d396f2a5f076603facde67f497bb69 WatchSource:0}: Error finding container f8e0f64348b2f0f7258fc78e1a688b3fa9d396f2a5f076603facde67f497bb69: Status 404 returned error can't find the container with id f8e0f64348b2f0f7258fc78e1a688b3fa9d396f2a5f076603facde67f497bb69
Oct 31 01:57:07.831068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.830986    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw" event={"ID":"5f6558d2-2bfa-4ace-af46-60e0c3b41415","Type":"ContainerStarted","Data":"c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b"}
Oct 31 01:57:07.831338 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.831315    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:07.831496 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.831354    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw" event={"ID":"5f6558d2-2bfa-4ace-af46-60e0c3b41415","Type":"ContainerStarted","Data":"f8e0f64348b2f0f7258fc78e1a688b3fa9d396f2a5f076603facde67f497bb69"}
Oct 31 01:57:07.833119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.833091    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" event={"ID":"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99","Type":"ContainerStarted","Data":"788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab"}
Oct 31 01:57:07.833241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.833153    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" event={"ID":"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99","Type":"ContainerStarted","Data":"d7f44ee69281692098487bb166fa6fd8d38678fa846cae9d7367fa6abab2d418"}
Oct 31 01:57:07.833241 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.833194    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:07.947180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:07.947109    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw" podStartSLOduration=1.9470915770000001 podStartE2EDuration="1.947091577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:07.943212737 +0000 UTC m=+3149.244169314" watchObservedRunningTime="2026-10-31 01:57:07.947091577 +0000 UTC m=+3149.248048157"
Oct 31 01:57:08.079174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:08.078215    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" podStartSLOduration=2.078191835 podStartE2EDuration="2.078191835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:08.056443907 +0000 UTC m=+3149.357400486" watchObservedRunningTime="2026-10-31 01:57:08.078191835 +0000 UTC m=+3149.379148413"
Oct 31 01:57:10.860460 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:10.860430    5419 generic.go:356] "Generic (PLEG): container finished" podID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerID="6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579" exitCode=0
Oct 31 01:57:10.860929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:10.860507    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" event={"ID":"38dc283f-bbcb-4544-8f81-13a0d10c52f6","Type":"ContainerDied","Data":"6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"}
Oct 31 01:57:10.860929 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:10.860885    5419 scope.go:122] "RemoveContainer" containerID="6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"
Oct 31 01:57:10.862317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:10.862291    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:10.862416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:10.862355    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:10.862416 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:10.862387    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:11.948073 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:11.948040    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" event={"ID":"38dc283f-bbcb-4544-8f81-13a0d10c52f6","Type":"ContainerStarted","Data":"78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3"}
Oct 31 01:57:12.274675 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:12.274642    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:57:12.275067 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:12.274959    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 31 01:57:12.275352 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:12.275328    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 31 01:57:13.257597 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:13.257557    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 31 01:57:13.258019 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:13.257632    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 31 01:57:14.257666 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:14.257631    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 31 01:57:14.258082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:14.257693    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 31 01:57:14.370442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:14.370411    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"]
Oct 31 01:57:14.373997 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:14.373976    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-cd0c98-6xvv2"]
Oct 31 01:57:14.632257 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:57:14.632172    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:57:14.840033 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:14.840003    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8" path="/var/lib/kubelet/pods/e99f3ecc-bbe5-4ad2-aeaa-f5f04686eba8/volumes"
Oct 31 01:57:15.256801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:15.256771    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 31 01:57:15.257010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:15.256825    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 31 01:57:16.257650 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:16.257613    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 31 01:57:16.258069 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:16.257671    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 31 01:57:17.256843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:17.256807    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused" start-of-body=
Oct 31 01:57:17.257044 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:17.256860    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.6:15021/healthz/ready\": dial tcp 10.133.1.6:15021: connect: connection refused"
Oct 31 01:57:18.262449 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:18.262417    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:57:18.262931 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:18.262911    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:57:18.263611 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:18.263596    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 01:57:18.846232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:18.846205    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 01:57:18.846403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:18.846266    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 01:57:20.808362 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:20.808315    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"]
Oct 31 01:57:20.808909 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:20.808565    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" containerID="cri-o://2699c7dc2540be3c9cc85cc469a55d16bcc75b2f07348be6f445bbb076aadc67" gracePeriod=30
Oct 31 01:57:22.098046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:22.098019    5419 generic.go:356] "Generic (PLEG): container finished" podID="689fb428-fba4-418f-b221-a7498437f213" containerID="b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d" exitCode=0
Oct 31 01:57:22.098476 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:22.098101    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" event={"ID":"689fb428-fba4-418f-b221-a7498437f213","Type":"ContainerDied","Data":"b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"}
Oct 31 01:57:22.098577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:22.098563    5419 scope.go:122] "RemoveContainer" containerID="b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"
Oct 31 01:57:22.100282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:22.100254    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:22.100396 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:22.100358    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:22.100466 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:22.100413    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:23.109430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:23.109387    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" event={"ID":"689fb428-fba4-418f-b221-a7498437f213","Type":"ContainerStarted","Data":"81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2"}
Oct 31 01:57:23.195160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:23.194585    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:57:23.195160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:23.194716    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 01:57:23.195160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:23.194776    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 01:57:24.199918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:24.199284    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 01:57:24.199918 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:24.199345    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 01:57:25.200191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:25.199522    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 01:57:25.200191 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:25.199584    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 01:57:25.713945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:25.713906    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"]
Oct 31 01:57:25.730823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:25.730779    5419 patch_prober.go:29] interesting pod/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:57:25.731012 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:25.730839    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:57:25.762799 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:25.762763    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3b7279-lbgq6"]
Oct 31 01:57:26.062090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.062022    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"]
Oct 31 01:57:26.067400 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.067367    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b6b4161-x77f8"]
Oct 31 01:57:26.166521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.166339    5419 generic.go:356] "Generic (PLEG): container finished" podID="e945423f-962d-4713-9925-08bc6f546dbe" containerID="2699c7dc2540be3c9cc85cc469a55d16bcc75b2f07348be6f445bbb076aadc67" exitCode=0
Oct 31 01:57:26.166521 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.166464    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" event={"ID":"e945423f-962d-4713-9925-08bc6f546dbe","Type":"ContainerDied","Data":"2699c7dc2540be3c9cc85cc469a55d16bcc75b2f07348be6f445bbb076aadc67"}
Oct 31 01:57:26.207882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.207648    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 01:57:26.207882 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.207705    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 01:57:26.302104 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.302074    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:57:26.362486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362392    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.362486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362461    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.362736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362530    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.362736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362554    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.362736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362595    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.362736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362642    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.362736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362697    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.363041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362741    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.363041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362762    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs\") pod \"e945423f-962d-4713-9925-08bc6f546dbe\" (UID: \"e945423f-962d-4713-9925-08bc6f546dbe\") "
Oct 31 01:57:26.363041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.362766    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.363041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.363012    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.363041 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.363022    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.363539 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.363394    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.364171 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.364111    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:26.364343 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.364172    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.364498 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.364480    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.364685 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.364664    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.364824 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.364808    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.366525 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.366499    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "kube-api-access-v85x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:26.367116 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.366991    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:57:26.367301 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.367235    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:26.368255 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.368220    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy" pod "e945423f-962d-4713-9925-08bc6f546dbe" (UID: "e945423f-962d-4713-9925-08bc6f546dbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:26.466190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.466159    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e945423f-962d-4713-9925-08bc6f546dbe-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.466190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.466186    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.466190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.466196    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e945423f-962d-4713-9925-08bc6f546dbe-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.466433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.466208    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e945423f-962d-4713-9925-08bc6f546dbe-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.466433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.466218    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v85x5\" (UniqueName: \"kubernetes.io/projected/e945423f-962d-4713-9925-08bc6f546dbe-kube-api-access-v85x5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:26.845316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.845279    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb784c2c-0fe4-4c40-8fe4-e9a5d5922001" path="/var/lib/kubelet/pods/bb784c2c-0fe4-4c40-8fe4-e9a5d5922001/volumes"
Oct 31 01:57:26.845825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:26.845808    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d4138c-e763-4909-803f-78068715bc55" path="/var/lib/kubelet/pods/f5d4138c-e763-4909-803f-78068715bc55/volumes"
Oct 31 01:57:27.178216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.175718    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d" event={"ID":"e945423f-962d-4713-9925-08bc6f546dbe","Type":"ContainerDied","Data":"fa395fa29c87acc1809b29c4edb9ca31b53ba4bc9bf2d9cc5e72d1b74d51c4d5"}
Oct 31 01:57:27.178216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.175794    5419 scope.go:122] "RemoveContainer" containerID="2699c7dc2540be3c9cc85cc469a55d16bcc75b2f07348be6f445bbb076aadc67"
Oct 31 01:57:27.178216 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.175961    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"
Oct 31 01:57:27.205281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.205241    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 01:57:27.205469 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.205320    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 01:57:27.346468 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.346432    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"]
Oct 31 01:57:27.404190 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:27.404131    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cd0c98-openshift-default-f9ccb5df7-dtr7d"]
Oct 31 01:57:28.193053 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:28.193016    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Oct 31 01:57:28.193280 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:28.193093    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Oct 31 01:57:28.838011 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:28.837977    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e945423f-962d-4713-9925-08bc6f546dbe" path="/var/lib/kubelet/pods/e945423f-962d-4713-9925-08bc6f546dbe/volumes"
Oct 31 01:57:29.197835 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:29.197763    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:57:29.198082 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:29.198062    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:57:29.198769 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:29.198750    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 01:57:30.046573 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:30.046536    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"]
Oct 31 01:57:30.095891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:30.095854    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb4ec3a-97vj2"]
Oct 31 01:57:30.838795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:30.838767    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0537df70-0add-4011-98a4-9bc3480b5933" path="/var/lib/kubelet/pods/0537df70-0add-4011-98a4-9bc3480b5933/volumes"
Oct 31 01:57:32.444348 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:32.444316    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"]
Oct 31 01:57:32.444854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:32.444581    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy" containerID="cri-o://3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949" gracePeriod=30
Oct 31 01:57:33.681541 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.681500    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"]
Oct 31 01:57:33.683367 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.683321    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy"
Oct 31 01:57:33.683546 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.683523    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy"
Oct 31 01:57:33.683945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.683928    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e945423f-962d-4713-9925-08bc6f546dbe" containerName="istio-proxy"
Oct 31 01:57:33.695540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.695515    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.767168 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.760982    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:57:33.791642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.791408    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:57:33.791642 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.791562    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:57:33.820554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.792331    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:57:33.820825 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.816025    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.821039 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.821015    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.821300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.821277    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.845786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.845745    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"]
Oct 31 01:57:33.874939 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.874905    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"]
Oct 31 01:57:33.881586 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.881559    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:33.925237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.923157    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.925237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.923203    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.925237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.923304    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.925237 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.924438    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.926643 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.926618    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:33.928212 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.928186    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"]
Oct 31 01:57:33.935846 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:33.935783    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c4d438-dockercfg-4g847\""
Oct 31 01:57:34.078324 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.077464    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c4d438\""
Oct 31 01:57:34.078774 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.078684    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf\") pod \"maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:34.108668 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.108454    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"]
Oct 31 01:57:34.122674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.122282    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vs9\" (UniqueName: \"kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9\") pod \"payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.122674 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.122460    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.127419 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.127393    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"]
Oct 31 01:57:34.130511 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.127671    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.132944 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.132916    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"]
Oct 31 01:57:34.133056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.132953    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"]
Oct 31 01:57:34.133056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.133032    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.197223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.187155    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"auth-test-model-5e916d-kserve-self-signed-certs\""
Oct 31 01:57:34.197223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.187273    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"openshift-service-ca.crt\""
Oct 31 01:57:34.197223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.187364    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"kube-root-ca.crt\""
Oct 31 01:57:34.197508 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.197427    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5e916d\"/\"default-dockercfg-tx2g9\""
Oct 31 01:57:34.227209 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227054    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.227430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227126    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.227430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227310    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.227430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227348    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.227430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227374    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mksn\" (UniqueName: \"kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.227430 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227399    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.227741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227531    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4r9\" (UniqueName: \"kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9\") pod \"payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.227741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227622    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.227741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227660    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vs9\" (UniqueName: \"kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9\") pod \"payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.227917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.227765    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.236457 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.228844    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.285644 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.285613    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vs9\" (UniqueName: \"kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9\") pod \"payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.316775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.316745    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:34.329200 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329169    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4r9\" (UniqueName: \"kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9\") pod \"payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.329328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329236    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.329328 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329266    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329341    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329371    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329416    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329449    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mksn\" (UniqueName: \"kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329608 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329480    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329805 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329771    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.329871 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.329848    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.330056 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.330030    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.330321 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.330300    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.331735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.331712    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.332889 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.332855    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.395775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.395745    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mksn\" (UniqueName: \"kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn\") pod \"auth-test-model-5e916d-kserve-57957cd989-8p7zd\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") " pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.397876 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.397851    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4r9\" (UniqueName: \"kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9\") pod \"payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.447405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.447333    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:34.453461 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.453438    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:34.505034 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.504750    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:34.704839 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.704803    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"]
Oct 31 01:57:34.705738 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:34.705703    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1770af60_2986_46b5_9b1f_13336aa12dc6.slice/crio-5233c49f805c692aa5ec2081989f2c5e206e8c34ed626ee11a3928205446a93f WatchSource:0}: Error finding container 5233c49f805c692aa5ec2081989f2c5e206e8c34ed626ee11a3928205446a93f: Status 404 returned error can't find the container with id 5233c49f805c692aa5ec2081989f2c5e206e8c34ed626ee11a3928205446a93f
Oct 31 01:57:34.914972 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:34.914945    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a74f67f_98a6_4ca1_92e1_987bdfe3024f.slice/crio-f3de448eb380dcb388fb656b4d862743b5ce570d8c78c017391804b3566c2f4f WatchSource:0}: Error finding container f3de448eb380dcb388fb656b4d862743b5ce570d8c78c017391804b3566c2f4f: Status 404 returned error can't find the container with id f3de448eb380dcb388fb656b4d862743b5ce570d8c78c017391804b3566c2f4f
Oct 31 01:57:34.915094 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.915051    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"]
Oct 31 01:57:34.963916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.963891    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"]
Oct 31 01:57:34.968600 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:34.968582    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"]
Oct 31 01:57:34.973635 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:34.973607    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde303017_0fcc_48aa_aaef_b3fcd150735b.slice/crio-be6cb640375cbac4b90b87fcc037578645f159774e80b657078522415becea8f WatchSource:0}: Error finding container be6cb640375cbac4b90b87fcc037578645f159774e80b657078522415becea8f: Status 404 returned error can't find the container with id be6cb640375cbac4b90b87fcc037578645f159774e80b657078522415becea8f
Oct 31 01:57:34.974035 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:34.974013    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9a1127_0e5b_4111_9e6e_3687ca47c6fb.slice/crio-ff31234caec0a0c19ef3ab24d012ea4fe1426682729a581f3f0eac191891f13a WatchSource:0}: Error finding container ff31234caec0a0c19ef3ab24d012ea4fe1426682729a581f3f0eac191891f13a: Status 404 returned error can't find the container with id ff31234caec0a0c19ef3ab24d012ea4fe1426682729a581f3f0eac191891f13a
Oct 31 01:57:35.254314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.254278    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c" event={"ID":"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb","Type":"ContainerStarted","Data":"4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e"}
Oct 31 01:57:35.254520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.254347    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c" event={"ID":"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb","Type":"ContainerStarted","Data":"ff31234caec0a0c19ef3ab24d012ea4fe1426682729a581f3f0eac191891f13a"}
Oct 31 01:57:35.254520 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.254379    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:35.256118 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.256094    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv" event={"ID":"7a74f67f-98a6-4ca1-92e1-987bdfe3024f","Type":"ContainerStarted","Data":"0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2"}
Oct 31 01:57:35.256265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.256150    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv" event={"ID":"7a74f67f-98a6-4ca1-92e1-987bdfe3024f","Type":"ContainerStarted","Data":"f3de448eb380dcb388fb656b4d862743b5ce570d8c78c017391804b3566c2f4f"}
Oct 31 01:57:35.256265 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.256200    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:35.257750 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.257723    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd" event={"ID":"de303017-0fcc-48aa-aaef-b3fcd150735b","Type":"ContainerStarted","Data":"4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18"}
Oct 31 01:57:35.257836 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.257758    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd" event={"ID":"de303017-0fcc-48aa-aaef-b3fcd150735b","Type":"ContainerStarted","Data":"be6cb640375cbac4b90b87fcc037578645f159774e80b657078522415becea8f"}
Oct 31 01:57:35.259187 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.259167    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5" event={"ID":"1770af60-2986-46b5-9b1f-13336aa12dc6","Type":"ContainerStarted","Data":"1eb0a1edd85c43223a4d965204c47d476fbdfaccf7199de43173ddb01bb319e0"}
Oct 31 01:57:35.259266 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.259205    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5" event={"ID":"1770af60-2986-46b5-9b1f-13336aa12dc6","Type":"ContainerStarted","Data":"5233c49f805c692aa5ec2081989f2c5e206e8c34ed626ee11a3928205446a93f"}
Oct 31 01:57:35.368329 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.368275    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c" podStartSLOduration=2.368256625 podStartE2EDuration="2.368256625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:35.366637852 +0000 UTC m=+3176.667594432" watchObservedRunningTime="2026-10-31 01:57:35.368256625 +0000 UTC m=+3176.669213202"
Oct 31 01:57:35.448008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.447973    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:35.495791 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.492675    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv" podStartSLOduration=2.492654177 podStartE2EDuration="2.492654177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:35.479287127 +0000 UTC m=+3176.780243762" watchObservedRunningTime="2026-10-31 01:57:35.492654177 +0000 UTC m=+3176.793610755"
Oct 31 01:57:35.735577 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.735526    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd" podStartSLOduration=2.7355100820000002 podStartE2EDuration="2.735510082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:35.733728962 +0000 UTC m=+3177.034685540" watchObservedRunningTime="2026-10-31 01:57:35.735510082 +0000 UTC m=+3177.036466716"
Oct 31 01:57:35.736519 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:35.736487    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5" podStartSLOduration=2.736476353 podStartE2EDuration="2.736476353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:35.620242055 +0000 UTC m=+3176.921198652" watchObservedRunningTime="2026-10-31 01:57:35.736476353 +0000 UTC m=+3177.037432929"
Oct 31 01:57:36.268602 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:36.268566    5419 generic.go:356] "Generic (PLEG): container finished" podID="1770af60-2986-46b5-9b1f-13336aa12dc6" containerID="1eb0a1edd85c43223a4d965204c47d476fbdfaccf7199de43173ddb01bb319e0" exitCode=0
Oct 31 01:57:36.268797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:36.268660    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5" event={"ID":"1770af60-2986-46b5-9b1f-13336aa12dc6","Type":"ContainerDied","Data":"1eb0a1edd85c43223a4d965204c47d476fbdfaccf7199de43173ddb01bb319e0"}
Oct 31 01:57:37.434771 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.434746    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:37.478207 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.478169    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf\") pod \"1770af60-2986-46b5-9b1f-13336aa12dc6\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") "
Oct 31 01:57:37.478347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.478230    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca\") pod \"1770af60-2986-46b5-9b1f-13336aa12dc6\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") "
Oct 31 01:57:37.478347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.478319    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca\") pod \"1770af60-2986-46b5-9b1f-13336aa12dc6\" (UID: \"1770af60-2986-46b5-9b1f-13336aa12dc6\") "
Oct 31 01:57:37.478729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.478701    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca" pod "1770af60-2986-46b5-9b1f-13336aa12dc6" (UID: "1770af60-2986-46b5-9b1f-13336aa12dc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:37.480556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.480532    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca" pod "1770af60-2986-46b5-9b1f-13336aa12dc6" (UID: "1770af60-2986-46b5-9b1f-13336aa12dc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:57:37.480556 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.480538    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf" pod "1770af60-2986-46b5-9b1f-13336aa12dc6" (UID: "1770af60-2986-46b5-9b1f-13336aa12dc6"). InnerVolumeSpecName "kube-api-access-lj8sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:37.580258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.580229    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8sf\" (UniqueName: \"kubernetes.io/projected/1770af60-2986-46b5-9b1f-13336aa12dc6-kube-api-access-lj8sf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.580258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.580255    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.580258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.580265    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1770af60-2986-46b5-9b1f-13336aa12dc6-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.702655 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.702623    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:57:37.729624 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.729603    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:57:37.781995 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.781964    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782041    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782076    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782114    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782191    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782222    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782302    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:37.782451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782317    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:37.782451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782321    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782451 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782424    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782495    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo\") pod \"f9a9a077-da45-490a-8148-6069b2f37d0c\" (UID: \"f9a9a077-da45-490a-8148-6069b2f37d0c\") "
Oct 31 01:57:37.782709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782519    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:37.782709 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782636    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:37.782913 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.782793    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:37.783406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.783384    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.783524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.783413    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.783524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.783427    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a9a077-da45-490a-8148-6069b2f37d0c-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.783524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.783442    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.783524 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.783456    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.784823 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.784803    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:37.784946 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.784924    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "kube-api-access-xmbf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:37.785016 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.784999    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:57:37.785211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.785189    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo" pod "f9a9a077-da45-490a-8148-6069b2f37d0c" (UID: "f9a9a077-da45-490a-8148-6069b2f37d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:57:37.885314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.885227    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.885314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.885258    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.885314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.885271    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a9a077-da45-490a-8148-6069b2f37d0c-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:37.885314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:37.885286    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmbf5\" (UniqueName: \"kubernetes.io/projected/f9a9a077-da45-490a-8148-6069b2f37d0c-kube-api-access-xmbf5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:38.285788 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.285763    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"
Oct 31 01:57:38.286010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.285762    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5" event={"ID":"1770af60-2986-46b5-9b1f-13336aa12dc6","Type":"ContainerDied","Data":"5233c49f805c692aa5ec2081989f2c5e206e8c34ed626ee11a3928205446a93f"}
Oct 31 01:57:38.286010 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.285893    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5233c49f805c692aa5ec2081989f2c5e206e8c34ed626ee11a3928205446a93f"
Oct 31 01:57:38.287801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.287773    5419 generic.go:356] "Generic (PLEG): container finished" podID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerID="3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949" exitCode=0
Oct 31 01:57:38.287968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.287811    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"
Oct 31 01:57:38.287968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.287851    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" event={"ID":"f9a9a077-da45-490a-8148-6069b2f37d0c","Type":"ContainerDied","Data":"3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949"}
Oct 31 01:57:38.287968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.287912    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn" event={"ID":"f9a9a077-da45-490a-8148-6069b2f37d0c","Type":"ContainerDied","Data":"0e472ea691f0b2193735e54612b6fdcce7dad3e53a276507a57266559dec915c"}
Oct 31 01:57:38.287968 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.287942    5419 scope.go:122] "RemoveContainer" containerID="3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949"
Oct 31 01:57:38.299869 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.299849    5419 scope.go:122] "RemoveContainer" containerID="b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"
Oct 31 01:57:38.310775 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.310754    5419 scope.go:122] "RemoveContainer" containerID="3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949"
Oct 31 01:57:38.311066 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:57:38.311048    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949\": container with ID starting with 3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949 not found: ID does not exist" containerID="3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949"
Oct 31 01:57:38.311113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.311075    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949"} err="failed to get container status \"3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949\": rpc error: code = NotFound desc = could not find container \"3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949\": container with ID starting with 3fca609edd6ee9da4582e7091352cd3296543bffbbedb1e3f84a1622fc1d4949 not found: ID does not exist"
Oct 31 01:57:38.311113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.311093    5419 scope.go:122] "RemoveContainer" containerID="b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"
Oct 31 01:57:38.311417 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:57:38.311397    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93\": container with ID starting with b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93 not found: ID does not exist" containerID="b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"
Oct 31 01:57:38.311473 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.311430    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93"} err="failed to get container status \"b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93\": rpc error: code = NotFound desc = could not find container \"b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93\": container with ID starting with b912fc14e9fcd6470b13739614f9a0dbd4fd8b022501eb948f2339fb46adff93 not found: ID does not exist"
Oct 31 01:57:38.453548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.453515    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"]
Oct 31 01:57:38.501866 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.501831    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-3b7279-openshift-default-c8b5c8dd5-8dpcn"]
Oct 31 01:57:38.838786 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:38.838758    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" path="/var/lib/kubelet/pods/f9a9a077-da45-490a-8148-6069b2f37d0c/volumes"
Oct 31 01:57:41.087185 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:41.087125    5419 scope.go:122] "RemoveContainer" containerID="307cd886cb6cef1f17683d8f8cc5ccfa45d4ae1d2a270b673a1aca17519eeffc"
Oct 31 01:57:41.097637 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:41.097614    5419 scope.go:122] "RemoveContainer" containerID="6f53fbbe5d1689a32f9a20e6091ea3f6dec209a471902198a16c2ca912d41fa0"
Oct 31 01:57:41.109023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:41.109004    5419 scope.go:122] "RemoveContainer" containerID="3104a2dae5d64b76aa5b233964fa9e8c1dad7434e099c3cef9dcc0da48ed73c5"
Oct 31 01:57:41.122277 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:41.122259    5419 scope.go:122] "RemoveContainer" containerID="327ff0cbf32457aefc8556ae830eccf087abf8ca5eb24e8dd7f4045b2cd1d97f"
Oct 31 01:57:44.047454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.047417    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"]
Oct 31 01:57:44.095355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.095308    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f62615d3-xlnbd"]
Oct 31 01:57:44.447952 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.447870    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:44.460166 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.460129    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:44.583251 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:57:44.583215    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:57:44.749226 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.749197    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"]
Oct 31 01:57:44.838720 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.838686    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8" path="/var/lib/kubelet/pods/5c24cdc9-ecdb-49fb-b5be-bf4ac41308d8/volumes"
Oct 31 01:57:44.855057 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:44.855030    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"]
Oct 31 01:57:45.353047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:45.353019    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 01:57:46.046981 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:46.046950    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"]
Oct 31 01:57:46.051436 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:46.051412    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57338433-wnjqd"]
Oct 31 01:57:46.297423 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:46.297346    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 01:57:46.298908 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:46.298885    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 01:57:46.838278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:46.838250    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06d5045a-a6a7-4c42-bcea-fa6b13cb7f18" path="/var/lib/kubelet/pods/06d5045a-a6a7-4c42-bcea-fa6b13cb7f18/volumes"
Oct 31 01:57:49.749927 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:49.749883    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl" podUID="fc76441f-c18a-4232-bf1d-e899ee0b4b3c" containerName="payload-processing" containerID="cri-o://6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a" gracePeriod=25
Oct 31 01:57:49.855540 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:49.855491    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf" podUID="68601002-b9af-4a6f-9968-33faf79bde28" containerName="payload-pre-processing" containerID="cri-o://3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2" gracePeriod=25
Oct 31 01:57:50.054702 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.047803    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:57:50.119052 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.119016    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5\" (UniqueName: \"kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5\") pod \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") "
Oct 31 01:57:50.119316 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.119213    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume\") pod \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\" (UID: \"fc76441f-c18a-4232-bf1d-e899ee0b4b3c\") "
Oct 31 01:57:50.119950 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.119831    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume" pod "fc76441f-c18a-4232-bf1d-e899ee0b4b3c" (UID: "fc76441f-c18a-4232-bf1d-e899ee0b4b3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:50.125355 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.125330    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5" pod "fc76441f-c18a-4232-bf1d-e899ee0b4b3c" (UID: "fc76441f-c18a-4232-bf1d-e899ee0b4b3c"). InnerVolumeSpecName "kube-api-access-5d4g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:50.155437 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.155414    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:57:50.254164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.247363    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume\") pod \"68601002-b9af-4a6f-9968-33faf79bde28\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") "
Oct 31 01:57:50.254164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.247556    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd\" (UniqueName: \"kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd\") pod \"68601002-b9af-4a6f-9968-33faf79bde28\" (UID: \"68601002-b9af-4a6f-9968-33faf79bde28\") "
Oct 31 01:57:50.254164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.248051    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:50.254164 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.248073    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4g5\" (UniqueName: \"kubernetes.io/projected/fc76441f-c18a-4232-bf1d-e899ee0b4b3c-kube-api-access-5d4g5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:50.303843 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.284372    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume" pod "68601002-b9af-4a6f-9968-33faf79bde28" (UID: "68601002-b9af-4a6f-9968-33faf79bde28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:57:50.338827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.338786    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd" pod "68601002-b9af-4a6f-9968-33faf79bde28" (UID: "68601002-b9af-4a6f-9968-33faf79bde28"). InnerVolumeSpecName "kube-api-access-ggdmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:57:50.349504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.349468    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggdmd\" (UniqueName: \"kubernetes.io/projected/68601002-b9af-4a6f-9968-33faf79bde28-kube-api-access-ggdmd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:50.349504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.349497    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68601002-b9af-4a6f-9968-33faf79bde28-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:57:50.386806 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.386766    5419 generic.go:356] "Generic (PLEG): container finished" podID="fc76441f-c18a-4232-bf1d-e899ee0b4b3c" containerID="6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a" exitCode=0
Oct 31 01:57:50.387014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.386813    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl" event={"ID":"fc76441f-c18a-4232-bf1d-e899ee0b4b3c","Type":"ContainerDied","Data":"6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a"}
Oct 31 01:57:50.387014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.386835    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"
Oct 31 01:57:50.387014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.386886    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl" event={"ID":"fc76441f-c18a-4232-bf1d-e899ee0b4b3c","Type":"ContainerDied","Data":"1534b4165e1053d449776598ab490b4748b015993be2e4145332a276eea69da4"}
Oct 31 01:57:50.387014 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.386917    5419 scope.go:122] "RemoveContainer" containerID="6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a"
Oct 31 01:57:50.388513 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.388492    5419 generic.go:356] "Generic (PLEG): container finished" podID="68601002-b9af-4a6f-9968-33faf79bde28" containerID="3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2" exitCode=0
Oct 31 01:57:50.388634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.388546    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"
Oct 31 01:57:50.388634 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.388581    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf" event={"ID":"68601002-b9af-4a6f-9968-33faf79bde28","Type":"ContainerDied","Data":"3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2"}
Oct 31 01:57:50.388724 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.388642    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf" event={"ID":"68601002-b9af-4a6f-9968-33faf79bde28","Type":"ContainerDied","Data":"c7d71fc06cf1399ff494989af9a58f72d37c192e6b2c8fe7b97d7412b262beb2"}
Oct 31 01:57:50.398278 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.398185    5419 scope.go:122] "RemoveContainer" containerID="6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a"
Oct 31 01:57:50.398543 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:57:50.398514    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a\": container with ID starting with 6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a not found: ID does not exist" containerID="6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a"
Oct 31 01:57:50.398656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.398551    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a"} err="failed to get container status \"6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a\": rpc error: code = NotFound desc = could not find container \"6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a\": container with ID starting with 6c90283144a7c31ad4aab99347de473816fca7f1a2509e861d8f33917ebfff9a not found: ID does not exist"
Oct 31 01:57:50.398656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.398571    5419 scope.go:122] "RemoveContainer" containerID="3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2"
Oct 31 01:57:50.409064 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.409040    5419 scope.go:122] "RemoveContainer" containerID="3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2"
Oct 31 01:57:50.409370 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:57:50.409350    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2\": container with ID starting with 3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2 not found: ID does not exist" containerID="3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2"
Oct 31 01:57:50.409447 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.409381    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2"} err="failed to get container status \"3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2\": rpc error: code = NotFound desc = could not find container \"3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2\": container with ID starting with 3321115f15f7bcb1482e7205c108ed1556ff6ae86be40f18bd41dd26f3a755b2 not found: ID does not exist"
Oct 31 01:57:50.475741 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.475715    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"]
Oct 31 01:57:50.476324 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476310    5419 cpu_manager.go:414] "containerMap: removing container" podUID="68601002-b9af-4a6f-9968-33faf79bde28" containerName="payload-pre-processing"
Oct 31 01:57:50.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476326    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68601002-b9af-4a6f-9968-33faf79bde28" containerName="payload-pre-processing"
Oct 31 01:57:50.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476341    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy"
Oct 31 01:57:50.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476347    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy"
Oct 31 01:57:50.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476354    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy"
Oct 31 01:57:50.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476360    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy"
Oct 31 01:57:50.476384 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476386    5419 cpu_manager.go:414] "containerMap: removing container" podUID="1770af60-2986-46b5-9b1f-13336aa12dc6" containerName="revoke-keys"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476391    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1770af60-2986-46b5-9b1f-13336aa12dc6" containerName="revoke-keys"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476400    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fc76441f-c18a-4232-bf1d-e899ee0b4b3c" containerName="payload-processing"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476404    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc76441f-c18a-4232-bf1d-e899ee0b4b3c" containerName="payload-processing"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476490    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="68601002-b9af-4a6f-9968-33faf79bde28" containerName="payload-pre-processing"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476498    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476503    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a9a077-da45-490a-8148-6069b2f37d0c" containerName="istio-proxy"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476513    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="1770af60-2986-46b5-9b1f-13336aa12dc6" containerName="revoke-keys"
Oct 31 01:57:50.476585 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.476519    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc76441f-c18a-4232-bf1d-e899ee0b4b3c" containerName="payload-processing"
Oct 31 01:57:50.481302 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.481283    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.536504 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.534219    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5632ea-openshift-default-dockercfg-v9rkv\""
Oct 31 01:57:50.540198 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.540172    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"]
Oct 31 01:57:50.551249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.551220    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"]
Oct 31 01:57:50.551845 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.551807    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.551921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.551864    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.551993 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.551967    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.552113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.552081    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.552225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.552164    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.552225 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.552222    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.552347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.552266    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.552347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.552289    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.552347 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.552319    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.557095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.557024    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-476ce2-5fb7b9b7f-qgwxl"]
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.672930    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.672500    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.673109    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.673791    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.673871    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.673979    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.674028    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.674103    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.676651    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.676702    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.676958 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.676897    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.678093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.677059    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.678093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.677546    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"]
Oct 31 01:57:50.678093 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.677711    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.678964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.678921    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.681194 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.680847    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.681596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.681528    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.685414 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.685385    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-476ce2-869447977f-cvzpf"]
Oct 31 01:57:50.732284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.732250    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.732425 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.732351    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz\") pod \"e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") " pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.794934 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.794907    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:50.933334 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.933292    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68601002-b9af-4a6f-9968-33faf79bde28" path="/var/lib/kubelet/pods/68601002-b9af-4a6f-9968-33faf79bde28/volumes"
Oct 31 01:57:50.933964 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:50.933938    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc76441f-c18a-4232-bf1d-e899ee0b4b3c" path="/var/lib/kubelet/pods/fc76441f-c18a-4232-bf1d-e899ee0b4b3c/volumes"
Oct 31 01:57:51.284879 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.284848    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"]
Oct 31 01:57:51.285989 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:51.285958    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606c61cc_b66d_44a3_adca_c194a4879ea9.slice/crio-b2725f6a22db1b6bbb1f22c32c7fc99a581a8506f3c8b9ef495dc71719977b0d WatchSource:0}: Error finding container b2725f6a22db1b6bbb1f22c32c7fc99a581a8506f3c8b9ef495dc71719977b0d: Status 404 returned error can't find the container with id b2725f6a22db1b6bbb1f22c32c7fc99a581a8506f3c8b9ef495dc71719977b0d
Oct 31 01:57:51.288639 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.288599    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:51.288735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.288675    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:51.288735 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.288708    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:57:51.403249 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.403209    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" event={"ID":"606c61cc-b66d-44a3-adca-c194a4879ea9","Type":"ContainerStarted","Data":"72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"}
Oct 31 01:57:51.403448 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.403278    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" event={"ID":"606c61cc-b66d-44a3-adca-c194a4879ea9","Type":"ContainerStarted","Data":"b2725f6a22db1b6bbb1f22c32c7fc99a581a8506f3c8b9ef495dc71719977b0d"}
Oct 31 01:57:51.521486 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.521436    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podStartSLOduration=1.521418567 podStartE2EDuration="1.521418567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:51.519041304 +0000 UTC m=+3192.819997885" watchObservedRunningTime="2026-10-31 01:57:51.521418567 +0000 UTC m=+3192.822375145"
Oct 31 01:57:51.795107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:51.795078    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:52.802083 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:52.802056    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:53.418441 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:53.418406    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:53.419314 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:53.419290    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:57:55.052692 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:55.052658    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"]
Oct 31 01:57:55.101649 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:55.101615    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e583f52-gz9bl"]
Oct 31 01:57:56.838102 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:56.838069    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a02b55-0131-4427-ac05-0c21235dde03" path="/var/lib/kubelet/pods/c4a02b55-0131-4427-ac05-0c21235dde03/volumes"
Oct 31 01:57:58.147160 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.144972    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"]
Oct 31 01:57:58.164607 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.164564    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.201180 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.197768    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"]
Oct 31 01:57:58.250480 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.250448    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5632ea\""
Oct 31 01:57:58.252043 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.251975    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5632ea-dockercfg-nbckf\""
Oct 31 01:57:58.288376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.288344    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmb2s\" (UniqueName: \"kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s\") pod \"payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.288551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.288455    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.389558 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.389520    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.389746 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.389729    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmb2s\" (UniqueName: \"kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s\") pod \"payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.390281 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.390263    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.446304 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.446240    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmb2s\" (UniqueName: \"kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s\") pod \"payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.527681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.527652    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:58.908372 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:58.908343    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"]
Oct 31 01:57:58.908877 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:57:58.908859    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb617360_e93d_4e60_be41_83bf1e6f8591.slice/crio-a514cc65c99bf1ca94a3635232333b6ab5ea9f50ebd94957ef568e28e02f3a33 WatchSource:0}: Error finding container a514cc65c99bf1ca94a3635232333b6ab5ea9f50ebd94957ef568e28e02f3a33: Status 404 returned error can't find the container with id a514cc65c99bf1ca94a3635232333b6ab5ea9f50ebd94957ef568e28e02f3a33
Oct 31 01:57:59.466891 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:59.466855    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2" event={"ID":"db617360-e93d-4e60-be41-83bf1e6f8591","Type":"ContainerStarted","Data":"80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15"}
Oct 31 01:57:59.467303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:59.466916    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2" event={"ID":"db617360-e93d-4e60-be41-83bf1e6f8591","Type":"ContainerStarted","Data":"a514cc65c99bf1ca94a3635232333b6ab5ea9f50ebd94957ef568e28e02f3a33"}
Oct 31 01:57:59.467303 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:59.466962    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:57:59.583366 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:57:59.583313    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2" podStartSLOduration=1.583298043 podStartE2EDuration="1.583298043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:59.580010301 +0000 UTC m=+3200.880966880" watchObservedRunningTime="2026-10-31 01:57:59.583298043 +0000 UTC m=+3200.884254642"
Oct 31 01:58:02.045464 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:02.045431    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"]
Oct 31 01:58:02.095008 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:02.094975    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a87a32a2-57bwh"]
Oct 31 01:58:02.838413 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:02.838385    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35c9b2f-e86a-48f0-9a87-79af6bfc6397" path="/var/lib/kubelet/pods/f35c9b2f-e86a-48f0-9a87-79af6bfc6397/volumes"
Oct 31 01:58:04.396684 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:04.396653    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:58:10.046105 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:10.046068    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"]
Oct 31 01:58:10.108638 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:10.108605    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a36982-gsw6c"]
Oct 31 01:58:10.477427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:10.477399    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 01:58:10.838284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:10.838211    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d4525f-8369-400b-bf1b-48c0d7b616f3" path="/var/lib/kubelet/pods/16d4525f-8369-400b-bf1b-48c0d7b616f3/volumes"
Oct 31 01:58:14.583340 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:58:14.583304    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:58:18.045721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:18.045684    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"]
Oct 31 01:58:18.078442 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:18.078408    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d13bcd5-r24gv"]
Oct 31 01:58:18.838534 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:18.838501    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d07ee7-e157-409a-9773-1ba958dd967e" path="/var/lib/kubelet/pods/61d07ee7-e157-409a-9773-1ba958dd967e/volumes"
Oct 31 01:58:27.047114 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:27.047066    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"]
Oct 31 01:58:27.050349 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:27.050325    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13843630-lp2nt"]
Oct 31 01:58:28.840682 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:28.840646    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178cc66b-3582-4bde-b580-948c67bb8a96" path="/var/lib/kubelet/pods/178cc66b-3582-4bde-b580-948c67bb8a96/volumes"
Oct 31 01:58:31.794300 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:31.794265    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:58:31.794712 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:31.794325    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:58:33.046719 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.046687    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"]
Oct 31 01:58:33.095921 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.095893    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523340b7-dmcw2"]
Oct 31 01:58:33.791240 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.791207    5419 generic.go:356] "Generic (PLEG): container finished" podID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerID="c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811" exitCode=0
Oct 31 01:58:33.791421 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.791282    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" event={"ID":"c8f89ea9-39c8-4263-92e4-34417a61dc0a","Type":"ContainerDied","Data":"c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"}
Oct 31 01:58:33.791752 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.791733    5419 scope.go:122] "RemoveContainer" containerID="c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"
Oct 31 01:58:33.793337 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.793307    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:58:33.793474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.793393    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:58:33.793474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:33.793443    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:58:34.807827 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:34.807782    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" event={"ID":"c8f89ea9-39c8-4263-92e4-34417a61dc0a","Type":"ContainerStarted","Data":"768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764"}
Oct 31 01:58:34.838858 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:34.838829    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0763834b-10a9-43f5-b308-1b1854a01ed9" path="/var/lib/kubelet/pods/0763834b-10a9-43f5-b308-1b1854a01ed9/volumes"
Oct 31 01:58:35.237285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:35.237258    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:58:35.237547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:35.237526    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Oct 31 01:58:35.237596 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:35.237576    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Oct 31 01:58:36.237849 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:36.237814    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Oct 31 01:58:36.238285 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:36.237865    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Oct 31 01:58:37.237262 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:37.237214    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Oct 31 01:58:37.237493 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:37.237299    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Oct 31 01:58:38.236940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:38.236901    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Oct 31 01:58:38.237392 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:38.236967    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Oct 31 01:58:39.237841 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:39.237799    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body=
Oct 31 01:58:39.238376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:39.237859    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused"
Oct 31 01:58:40.240965 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:40.240932    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:58:40.241510 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:40.241373    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:58:40.242096 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:40.242075    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 01:58:41.175672 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:41.175638    5419 scope.go:122] "RemoveContainer" containerID="b0e24bc6cb7a74236ad4aa0198e204c4c2beb040e22427af99a5482a60b00c69"
Oct 31 01:58:41.187046 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:41.187014    5419 scope.go:122] "RemoveContainer" containerID="73bffe3bdfa53c75154c5c35356b7695566b14d2299f0c96bdf88b21fc36dd67"
Oct 31 01:58:41.196376 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:41.196359    5419 scope.go:122] "RemoveContainer" containerID="0c4b539b064370f160e2de7bf2f1428ebe2edf9b8126964ba9f2ee05bfe59ccc"
Oct 31 01:58:41.209211 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:41.209193    5419 scope.go:122] "RemoveContainer" containerID="4c9d0735a39b062d4a37dd7597a2300a7b06e967e89a3e4e614df9d93bda3c73"
Oct 31 01:58:41.218427 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:41.218404    5419 scope.go:122] "RemoveContainer" containerID="e1ffd0522a26427068c7a4e2a0374690cffc74d846ead12bcce419b5a94e641f"
Oct 31 01:58:44.046401 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:44.046366    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"]
Oct 31 01:58:44.052192 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:44.050238    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bac52fa-h92b5"]
Oct 31 01:58:44.625368 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:58:44.625326    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:58:44.838883 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:44.838852    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e60f2b84-b55e-4b1a-a826-4a630ef10a55" path="/var/lib/kubelet/pods/e60f2b84-b55e-4b1a-a826-4a630ef10a55/volumes"
Oct 31 01:58:46.141716 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:46.141678    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"]
Oct 31 01:58:46.191119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:46.191087    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-476ce2-h58z5"]
Oct 31 01:58:46.838312 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:46.838279    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1770af60-2986-46b5-9b1f-13336aa12dc6" path="/var/lib/kubelet/pods/1770af60-2986-46b5-9b1f-13336aa12dc6/volumes"
Oct 31 01:58:47.047271 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:47.047238    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"]
Oct 31 01:58:47.095653 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:47.095556    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f7313c2-prfrh"]
Oct 31 01:58:48.838284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:48.838252    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17153e24-fc3f-4582-b071-68db4cfdf0cf" path="/var/lib/kubelet/pods/17153e24-fc3f-4582-b071-68db4cfdf0cf/volumes"
Oct 31 01:58:54.599729 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:54.599693    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:58:59.046548 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:59.046516    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"]
Oct 31 01:58:59.095853 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:58:59.095822    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d32f34-cslrx"]
Oct 31 01:59:00.843529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:00.840662    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93bde77-e0be-4cf5-85fa-4c1b8c097f42" path="/var/lib/kubelet/pods/a93bde77-e0be-4cf5-85fa-4c1b8c097f42/volumes"
Oct 31 01:59:01.045721 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:01.045695    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"]
Oct 31 01:59:01.093665 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:01.093589    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d3eaeb2-ttwhk"]
Oct 31 01:59:02.837904 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:02.837874    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e76f6a-b47a-43f3-8c93-583d8b3e45d9" path="/var/lib/kubelet/pods/75e76f6a-b47a-43f3-8c93-583d8b3e45d9/volumes"
Oct 31 01:59:03.047537 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:03.047502    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"]
Oct 31 01:59:03.096412 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:03.096340    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4191a066-xwkn6"]
Oct 31 01:59:04.839080 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:04.839046    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6" path="/var/lib/kubelet/pods/bd8a23e0-2780-4c52-ad0b-0a7ad66c68a6/volumes"
Oct 31 01:59:10.047202 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:10.047164    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"]
Oct 31 01:59:10.078790 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:10.078762    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bacb09ee-r6l99"]
Oct 31 01:59:10.839465 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:10.839433    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48bca69-e103-43f6-9757-fe8a87f1b90d" path="/var/lib/kubelet/pods/d48bca69-e103-43f6-9757-fe8a87f1b90d/volumes"
Oct 31 01:59:14.583222 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:59:14.583185    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:59:21.050113 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:21.050080    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"]
Oct 31 01:59:21.098317 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:21.098282    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43c0d396-fxcts"]
Oct 31 01:59:22.837795 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:22.837757    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be41142d-f8dc-4241-8378-df71319b8a4e" path="/var/lib/kubelet/pods/be41142d-f8dc-4241-8378-df71319b8a4e/volumes"
Oct 31 01:59:25.075162 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:25.068218    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"]
Oct 31 01:59:25.120175 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:25.119543    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df862bdf-lcp6v"]
Oct 31 01:59:25.411810 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:25.411729    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 01:59:26.047038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:26.047002    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"]
Oct 31 01:59:26.094373 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:26.094337    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d2a4a5-srrsh"]
Oct 31 01:59:26.839758 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:26.839719    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba5e8b9e-ce60-4185-98e7-333dfac58b56" path="/var/lib/kubelet/pods/ba5e8b9e-ce60-4185-98e7-333dfac58b56/volumes"
Oct 31 01:59:26.840322 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:26.840299    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6bf222-b732-48e6-a49a-e28d7ee979c5" path="/var/lib/kubelet/pods/cc6bf222-b732-48e6-a49a-e28d7ee979c5/volumes"
Oct 31 01:59:32.045038 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:32.044977    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"]
Oct 31 01:59:32.093739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:32.093704    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca630013-c29ch"]
Oct 31 01:59:32.840881 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:32.840845    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39f7ee8-b620-4450-8c06-ade4d275b19a" path="/var/lib/kubelet/pods/d39f7ee8-b620-4450-8c06-ade4d275b19a/volumes"
Oct 31 01:59:35.354989 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:35.354909    5419 generic.go:356] "Generic (PLEG): container finished" podID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerID="72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c" exitCode=0
Oct 31 01:59:35.355492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:35.355020    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" event={"ID":"606c61cc-b66d-44a3-adca-c194a4879ea9","Type":"ContainerDied","Data":"72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"}
Oct 31 01:59:35.355492 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:35.355470    5419 scope.go:122] "RemoveContainer" containerID="72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"
Oct 31 01:59:35.357063 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:35.357027    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:59:35.357119 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:35.357099    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:59:35.357178 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:35.357130    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 01:59:36.364273 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.364241    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" event={"ID":"606c61cc-b66d-44a3-adca-c194a4879ea9","Type":"ContainerStarted","Data":"7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e"}
Oct 31 01:59:36.399199 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.374761    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"]
Oct 31 01:59:36.400487 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.400460    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.433854 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.433814    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"]
Oct 31 01:59:36.452307 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.452276    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:59:36.465389 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.465352    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:59:36.465568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.465408    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:59:36.465568 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.465410    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:59:36.504506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.504465    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.504743 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.504646    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grl82\" (UniqueName: \"kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.504817 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.504762    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.606021 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.605982    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.606263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.606096    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grl82\" (UniqueName: \"kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.606263 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.606231    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.606801 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.606776    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.609090 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.609060    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.662706 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.662637    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grl82\" (UniqueName: \"kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82\") pod \"maas-api-revoke-sub-fded139e-2nhr2\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.714284 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.714250    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:36.804286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.801954    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:59:36.804286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.803584    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 01:59:36.804286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:36.803633    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 01:59:37.109453 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:37.109426    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"]
Oct 31 01:59:37.111775 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:59:37.111744    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a3f596_0287_4780_aac0_76fe95d3b22e.slice/crio-c0496128fef587d3a5d13d5094ce2b2cc3f9da4336f321bdae95eebbe7f0be25 WatchSource:0}: Error finding container c0496128fef587d3a5d13d5094ce2b2cc3f9da4336f321bdae95eebbe7f0be25: Status 404 returned error can't find the container with id c0496128fef587d3a5d13d5094ce2b2cc3f9da4336f321bdae95eebbe7f0be25
Oct 31 01:59:37.373232 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:37.373129    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2" event={"ID":"07a3f596-0287-4780-aac0-76fe95d3b22e","Type":"ContainerStarted","Data":"c0496128fef587d3a5d13d5094ce2b2cc3f9da4336f321bdae95eebbe7f0be25"}
Oct 31 01:59:37.796730 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:37.796693    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 01:59:37.797018 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:37.796990    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 01:59:38.406912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:38.406821    5419 generic.go:356] "Generic (PLEG): container finished" podID="07a3f596-0287-4780-aac0-76fe95d3b22e" containerID="30a61eb097092afdb92a2004a90aabe791982381354f7cd4551ebeb035e631aa" exitCode=0
Oct 31 01:59:38.406912 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:38.406896    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2" event={"ID":"07a3f596-0287-4780-aac0-76fe95d3b22e","Type":"ContainerDied","Data":"30a61eb097092afdb92a2004a90aabe791982381354f7cd4551ebeb035e631aa"}
Oct 31 01:59:38.418945 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:38.418915    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:59:38.796603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:38.796568    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 01:59:38.796940 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:38.796914    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 01:59:39.624698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.624672    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:39.749258 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.749214    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca\") pod \"07a3f596-0287-4780-aac0-76fe95d3b22e\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") "
Oct 31 01:59:39.749454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.749377    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82\" (UniqueName: \"kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82\") pod \"07a3f596-0287-4780-aac0-76fe95d3b22e\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") "
Oct 31 01:59:39.749454 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.749430    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca\") pod \"07a3f596-0287-4780-aac0-76fe95d3b22e\" (UID: \"07a3f596-0287-4780-aac0-76fe95d3b22e\") "
Oct 31 01:59:39.749603 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.749573    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca" pod "07a3f596-0287-4780-aac0-76fe95d3b22e" (UID: "07a3f596-0287-4780-aac0-76fe95d3b22e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:39.750023 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.750004    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:39.751656 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.751633    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82" pod "07a3f596-0287-4780-aac0-76fe95d3b22e" (UID: "07a3f596-0287-4780-aac0-76fe95d3b22e"). InnerVolumeSpecName "kube-api-access-grl82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:39.751737 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.751664    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca" pod "07a3f596-0287-4780-aac0-76fe95d3b22e" (UID: "07a3f596-0287-4780-aac0-76fe95d3b22e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:59:39.801399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.800047    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 01:59:39.801399 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.800123    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 01:59:39.851380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.851343    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grl82\" (UniqueName: \"kubernetes.io/projected/07a3f596-0287-4780-aac0-76fe95d3b22e-kube-api-access-grl82\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:39.851380 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:39.851374    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a3f596-0287-4780-aac0-76fe95d3b22e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:40.422358 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:40.422330    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"
Oct 31 01:59:40.422545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:40.422354    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2" event={"ID":"07a3f596-0287-4780-aac0-76fe95d3b22e","Type":"ContainerDied","Data":"c0496128fef587d3a5d13d5094ce2b2cc3f9da4336f321bdae95eebbe7f0be25"}
Oct 31 01:59:40.422545 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:40.422410    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0496128fef587d3a5d13d5094ce2b2cc3f9da4336f321bdae95eebbe7f0be25"
Oct 31 01:59:40.799529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:40.799273    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 31 01:59:40.799529 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:40.799343    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 31 01:59:41.008536 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.008499    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"]
Oct 31 01:59:41.008773 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.008701    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy" containerID="cri-o://58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304" gracePeriod=30
Oct 31 01:59:41.293916 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.293878    5419 scope.go:122] "RemoveContainer" containerID="52b5afd75bcf5cf71a3e4c7a11162789a48e69e639bdd6a24a4aba3d65e3a126"
Oct 31 01:59:41.304739 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.304712    5419 scope.go:122] "RemoveContainer" containerID="c5ce03a8763e062ef9cb4c07d0f07f775f6558f5682258ec339f47e2c32ab6c4"
Oct 31 01:59:41.314606 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.314575    5419 scope.go:122] "RemoveContainer" containerID="17004a7385addae8a0983532c3c8022edaf7c6538607bc5acc8623a6f444beb2"
Oct 31 01:59:41.326109 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.326089    5419 scope.go:122] "RemoveContainer" containerID="b5744c5390d2ee57c3a404fa0c61dbdaf66533c1aca9428bccfc7efbd6959d1e"
Oct 31 01:59:41.335698 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.335677    5419 scope.go:122] "RemoveContainer" containerID="45c5bc4111e0e591d75b67f608727da66655e5afbe136f811cb4508e9d6f501e"
Oct 31 01:59:41.345346 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.345327    5419 scope.go:122] "RemoveContainer" containerID="36c5c0a012c1bb4b2878073070f2a5eea9a3e4a1340072a21f0bdf0c068acae2"
Oct 31 01:59:41.354572 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.354555    5419 scope.go:122] "RemoveContainer" containerID="593c93c16ea1df1186b1ba68164eec363155ff15528530402707f7db2e2d4bb7"
Oct 31 01:59:41.363588 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.363566    5419 scope.go:122] "RemoveContainer" containerID="48a282bd93930f4eab72517e1f9732e69184ac9f1c79273a42ea23a018e306c5"
Oct 31 01:59:41.372907 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.372886    5419 scope.go:122] "RemoveContainer" containerID="5356138bb8d7643a115b01a8dc467ee0fe4a83a595e6356a85f614f4dbd5db71"
Oct 31 01:59:41.800467 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.800435    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:59:41.801736 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:41.801718    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 01:59:44.587024 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:59:44.586983    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 01:59:46.287738 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.287674    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:59:46.460777 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.460743    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.460959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.460814    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.460959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.460881    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.460959 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.460940    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.461107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.460979    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.461107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.460999    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.461107 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461044    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.461341 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461192    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:46.461418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461328    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:46.461418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461313    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:46.461418 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461376    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:46.461578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461476    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.461578 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.461506    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data\") pod \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\" (UID: \"7faebc63-2f55-4fc6-b159-2a462ecdffd2\") "
Oct 31 01:59:46.462047 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.462020    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:46.462201 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.462181    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.462286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.462208    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.462286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.462224    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.462286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.462240    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.462286 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.462255    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.463403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.463371    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:46.463565 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.463534    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "kube-api-access-m9k59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:46.463754 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.463734    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:59:46.463813 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.463769    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy" pod "7faebc63-2f55-4fc6-b159-2a462ecdffd2" (UID: "7faebc63-2f55-4fc6-b159-2a462ecdffd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:59:46.475403 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.475375    5419 generic.go:356] "Generic (PLEG): container finished" podID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerID="58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304" exitCode=0
Oct 31 01:59:46.475506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.475413    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" event={"ID":"7faebc63-2f55-4fc6-b159-2a462ecdffd2","Type":"ContainerDied","Data":"58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304"}
Oct 31 01:59:46.475506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.475434    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"
Oct 31 01:59:46.475506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.475477    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p" event={"ID":"7faebc63-2f55-4fc6-b159-2a462ecdffd2","Type":"ContainerDied","Data":"145acac5b051bfd5e53356437bd075c7580b26c5b5ea6772bd77f28d13bde357"}
Oct 31 01:59:46.475506 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.475503    5419 scope.go:122] "RemoveContainer" containerID="58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304"
Oct 31 01:59:46.495309 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.495293    5419 scope.go:122] "RemoveContainer" containerID="58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304"
Oct 31 01:59:46.495623 ip-10-0-140-164 kubenswrapper[5419]: E1031 01:59:46.495601    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304\": container with ID starting with 58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304 not found: ID does not exist" containerID="58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304"
Oct 31 01:59:46.495681 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.495630    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304"} err="failed to get container status \"58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304\": rpc error: code = NotFound desc = could not find container \"58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304\": container with ID starting with 58ecf8a66ffb54be8f3b73dc6d90c7c3c66577ae530e8032f4442ecd6b701304 not found: ID does not exist"
Oct 31 01:59:46.563762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.563685    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.563762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.563714    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.563762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.563724    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9k59\" (UniqueName: \"kubernetes.io/projected/7faebc63-2f55-4fc6-b159-2a462ecdffd2-kube-api-access-m9k59\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.563762 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.563734    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7faebc63-2f55-4fc6-b159-2a462ecdffd2-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:46.638648 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.638614    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"]
Oct 31 01:59:46.642219 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.642195    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-xzn2p"]
Oct 31 01:59:46.838407 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:46.838322    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" path="/var/lib/kubelet/pods/7faebc63-2f55-4fc6-b159-2a462ecdffd2/volumes"
Oct 31 01:59:47.066095 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066062    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"]
Oct 31 01:59:47.066723 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066709    5419 cpu_manager.go:414] "containerMap: removing container" podUID="07a3f596-0287-4780-aac0-76fe95d3b22e" containerName="revoke-keys"
Oct 31 01:59:47.066797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066724    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a3f596-0287-4780-aac0-76fe95d3b22e" containerName="revoke-keys"
Oct 31 01:59:47.066797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066740    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy"
Oct 31 01:59:47.066797 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066745    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy"
Oct 31 01:59:47.066896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066839    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a3f596-0287-4780-aac0-76fe95d3b22e" containerName="revoke-keys"
Oct 31 01:59:47.066896 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.066848    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7faebc63-2f55-4fc6-b159-2a462ecdffd2" containerName="istio-proxy"
Oct 31 01:59:47.071547 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.071524    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.124177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.124066    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 01:59:47.124177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.124108    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 01:59:47.124177 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.124127    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 01:59:47.124405 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.124217    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 01:59:47.127068 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.127046    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"]
Oct 31 01:59:47.171474 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.171444    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvz7t\" (UniqueName: \"kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.171688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.171484    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.171688 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.171618    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.272551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.272508    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvz7t\" (UniqueName: \"kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.272551 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.272556    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.272860 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.272614    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.273335 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.273312    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.275106 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.275086    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.329917 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.329883    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvz7t\" (UniqueName: \"kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t\") pod \"maas-api-revoke-sub-e3140eac-2n8lt\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.384554 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.384476    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:47.759646 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:47.759617    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"]
Oct 31 01:59:47.760305 ip-10-0-140-164 kubenswrapper[5419]: W1031 01:59:47.760270    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e774ae5_b8ae_4ad3_a5aa_8628814003a0.slice/crio-f2b48b581580c233e0c0671f0398d5362e864cd3e71862c3664a10421e0793c2 WatchSource:0}: Error finding container f2b48b581580c233e0c0671f0398d5362e864cd3e71862c3664a10421e0793c2: Status 404 returned error can't find the container with id f2b48b581580c233e0c0671f0398d5362e864cd3e71862c3664a10421e0793c2
Oct 31 01:59:48.502796 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:48.502762    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt" event={"ID":"4e774ae5-b8ae-4ad3-a5aa-8628814003a0","Type":"ContainerStarted","Data":"c0740d0e5f747aa3dc8a60823a898205b7390da919f318a2187cc23d9e8d7218"}
Oct 31 01:59:48.503205 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:48.502814    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt" event={"ID":"4e774ae5-b8ae-4ad3-a5aa-8628814003a0","Type":"ContainerStarted","Data":"f2b48b581580c233e0c0671f0398d5362e864cd3e71862c3664a10421e0793c2"}
Oct 31 01:59:48.668483 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:48.668433    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt" podStartSLOduration=1.6684181809999998 podStartE2EDuration="1.668418181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:48.617220745 +0000 UTC m=+3309.918177323" watchObservedRunningTime="2026-10-31 01:59:48.668418181 +0000 UTC m=+3309.969374771"
Oct 31 01:59:49.511749 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:49.511718    5419 generic.go:356] "Generic (PLEG): container finished" podID="4e774ae5-b8ae-4ad3-a5aa-8628814003a0" containerID="c0740d0e5f747aa3dc8a60823a898205b7390da919f318a2187cc23d9e8d7218" exitCode=0
Oct 31 01:59:49.512174 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:49.511810    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt" event={"ID":"4e774ae5-b8ae-4ad3-a5aa-8628814003a0","Type":"ContainerDied","Data":"c0740d0e5f747aa3dc8a60823a898205b7390da919f318a2187cc23d9e8d7218"}
Oct 31 01:59:50.047433 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.047396    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"]
Oct 31 01:59:50.050159 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.050119    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d9e3794-qcf7v"]
Oct 31 01:59:50.675184 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.675161    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 01:59:50.733060 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.733028    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca\") pod \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") "
Oct 31 01:59:50.733282 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.733191    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t\" (UniqueName: \"kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t\") pod \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") "
Oct 31 01:59:50.733406 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.733391    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca\") pod \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\" (UID: \"4e774ae5-b8ae-4ad3-a5aa-8628814003a0\") "
Oct 31 01:59:50.733755 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.733724    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca" pod "4e774ae5-b8ae-4ad3-a5aa-8628814003a0" (UID: "4e774ae5-b8ae-4ad3-a5aa-8628814003a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:59:50.733868 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.733852    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:50.735320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.735298    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t" pod "4e774ae5-b8ae-4ad3-a5aa-8628814003a0" (UID: "4e774ae5-b8ae-4ad3-a5aa-8628814003a0"). InnerVolumeSpecName "kube-api-access-gvz7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:59:50.735320 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.735303    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca" pod "4e774ae5-b8ae-4ad3-a5aa-8628814003a0" (UID: "4e774ae5-b8ae-4ad3-a5aa-8628814003a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:59:50.835523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.835429    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:50.835523 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.835465    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvz7t\" (UniqueName: \"kubernetes.io/projected/4e774ae5-b8ae-4ad3-a5aa-8628814003a0-kube-api-access-gvz7t\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:59:50.839000 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:50.838973    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549a4e9b-0fef-4712-9941-ed5a2e51a0b9" path="/var/lib/kubelet/pods/549a4e9b-0fef-4712-9941-ed5a2e51a0b9/volumes"
Oct 31 01:59:51.528223 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:51.528182    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt" event={"ID":"4e774ae5-b8ae-4ad3-a5aa-8628814003a0","Type":"ContainerDied","Data":"f2b48b581580c233e0c0671f0398d5362e864cd3e71862c3664a10421e0793c2"}
Oct 31 01:59:51.528391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:51.528278    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b48b581580c233e0c0671f0398d5362e864cd3e71862c3664a10421e0793c2"
Oct 31 01:59:51.528391 ip-10-0-140-164 kubenswrapper[5419]: I1031 01:59:51.528199    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"
Oct 31 02:00:00.159362 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.159314    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"]
Oct 31 02:00:00.159985 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.159958    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4e774ae5-b8ae-4ad3-a5aa-8628814003a0" containerName="revoke-keys"
Oct 31 02:00:00.159985 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.159978    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e774ae5-b8ae-4ad3-a5aa-8628814003a0" containerName="revoke-keys"
Oct 31 02:00:00.160098 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.160088    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e774ae5-b8ae-4ad3-a5aa-8628814003a0" containerName="revoke-keys"
Oct 31 02:00:00.163759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.163742    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"]
Oct 31 02:00:00.163870 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.163854    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.167666 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.167644    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"]
Oct 31 02:00:00.167806 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.167750    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.171661 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.171642    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.208053 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.207979    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"]
Oct 31 02:00:00.213369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.213342    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"]
Oct 31 02:00:00.219542 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.219509    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"]
Oct 31 02:00:00.221076 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.221051    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.221298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.221246    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.225187 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.225162    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"]
Oct 31 02:00:00.225310 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.225265    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.229734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.229711    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"]
Oct 31 02:00:00.229838 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.229816    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.234770 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.234754    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"]
Oct 31 02:00:00.234871 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.234838    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.238926 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.238910    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.254420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.254401    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:00:00.254570 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.254437    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:00:00.254651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.254587    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:00:00.259300 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259276    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jgg\" (UniqueName: \"kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.259408 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259332    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx22d\" (UniqueName: \"kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.259480 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259434    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.259480 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259469    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.259607 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259561    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.259607 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259588    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.259732 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259619    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-lw8tv\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.259732 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259702    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj9ns\" (UniqueName: \"kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.259853 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259753    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85tpl\" (UniqueName: \"kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl\") pod \"maas-api-key-cleanup-29890200-lw8tv\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.259853 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259782    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.259853 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259807    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"]
Oct 31 02:00:00.259853 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259845    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59pfk\" (UniqueName: \"kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.260082 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.259911    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcdv\" (UniqueName: \"kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.262748 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.262727    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"]
Oct 31 02:00:00.265431 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.265414    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"]
Oct 31 02:00:00.310115 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.309974    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"]
Oct 31 02:00:00.313116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.313091    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"]
Oct 31 02:00:00.317894 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.317868    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"]
Oct 31 02:00:00.361539 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361478    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcdv\" (UniqueName: \"kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.361539 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361545    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jgg\" (UniqueName: \"kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.361827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361597    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx22d\" (UniqueName: \"kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.361827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361645    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfvt\" (UniqueName: \"kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.361827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361683    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.361827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361720    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbltm\" (UniqueName: \"kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.361827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361748    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.361827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361796    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361868    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361908    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361946    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbdm\" (UniqueName: \"kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.361979    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-lw8tv\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362014    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj9ns\" (UniqueName: \"kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362061    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362119    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tpl\" (UniqueName: \"kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl\") pod \"maas-api-key-cleanup-29890200-lw8tv\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362178    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362209    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59pfk\" (UniqueName: \"kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.362369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362261    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.362900 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.362624    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.363182 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.363078    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.363399 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.363378    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.363807 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.363781    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.363964 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.363864    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.364427 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.364405    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-lw8tv\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.367597 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.367577    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"]
Oct 31 02:00:00.370783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.370766    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"]
Oct 31 02:00:00.383720 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.383695    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"]
Oct 31 02:00:00.419043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.418842    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jgg\" (UniqueName: \"kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.419043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.418929    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcdv\" (UniqueName: \"kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv\") pod \"maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.419328 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.419309    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59pfk\" (UniqueName: \"kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.419443 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.419379    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx22d\" (UniqueName: \"kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d\") pod \"maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.419659 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.419638    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj9ns\" (UniqueName: \"kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.419841 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.419818    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tpl\" (UniqueName: \"kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl\") pod \"maas-api-key-cleanup-29890200-lw8tv\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.463792 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.463759    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.463949 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.463866    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfvt\" (UniqueName: \"kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.463949 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.463917    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbltm\" (UniqueName: \"kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.464052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.463967    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.464112 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.464059    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbdm\" (UniqueName: \"kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.464511 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.464121    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.464586 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.464564    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.464769 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.464751    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.464810 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.464776    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.479276 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.479257    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:00.487494 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.487477    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:00.493542 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.493526    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:00.521054 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.521023    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfvt\" (UniqueName: \"kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt\") pod \"maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.521257 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.521025    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbdm\" (UniqueName: \"kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.522409 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.522364    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbltm\" (UniqueName: \"kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm\") pod \"maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.542387 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.542349    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:00.547333 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.547106    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:00.575074 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.573416    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:00.575074 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.573822    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:00.575074 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.574959    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:00.578117 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.578093    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:00.868623 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:00.868576    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"]
Oct 31 02:00:00.873098 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:00.873050    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49161a2_5532_4906_92d5_33d4fd8cbc86.slice/crio-975293acf943a2c022d9520021f0f8493b292bca9e7b084e6eaeee26f94de5dc WatchSource:0}: Error finding container 975293acf943a2c022d9520021f0f8493b292bca9e7b084e6eaeee26f94de5dc: Status 404 returned error can't find the container with id 975293acf943a2c022d9520021f0f8493b292bca9e7b084e6eaeee26f94de5dc
Oct 31 02:00:01.027973 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.027951    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"]
Oct 31 02:00:01.029616 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:01.029591    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69de7d1_e1d7_4d5d_89c9_606556d6f7ec.slice/crio-520400e27c1b5abc0a386f4da3ab7662f9268c00fa51b87e03ba327edc1e1614 WatchSource:0}: Error finding container 520400e27c1b5abc0a386f4da3ab7662f9268c00fa51b87e03ba327edc1e1614: Status 404 returned error can't find the container with id 520400e27c1b5abc0a386f4da3ab7662f9268c00fa51b87e03ba327edc1e1614
Oct 31 02:00:01.079641 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.078493    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"]
Oct 31 02:00:01.083993 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:01.083968    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4b1214_a89c_4107_b7d0_971463d0c16f.slice/crio-99300b0b87efb92efa45f985e5ab52969d233215fa1812631a4c40bf6068fa07 WatchSource:0}: Error finding container 99300b0b87efb92efa45f985e5ab52969d233215fa1812631a4c40bf6068fa07: Status 404 returned error can't find the container with id 99300b0b87efb92efa45f985e5ab52969d233215fa1812631a4c40bf6068fa07
Oct 31 02:00:01.085174 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:01.085154    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f5da2d_3a9a_4e1e_bd12_e0847f2b638d.slice/crio-516cc95e8ed5a03a23c11aecf9a8d785d0ff4b2cdf6522419f6afa326610b4b6 WatchSource:0}: Error finding container 516cc95e8ed5a03a23c11aecf9a8d785d0ff4b2cdf6522419f6afa326610b4b6: Status 404 returned error can't find the container with id 516cc95e8ed5a03a23c11aecf9a8d785d0ff4b2cdf6522419f6afa326610b4b6
Oct 31 02:00:01.087266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.085466    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"]
Oct 31 02:00:01.089579 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.089304    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"]
Oct 31 02:00:01.092300 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.092280    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"]
Oct 31 02:00:01.095230 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.095214    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"]
Oct 31 02:00:01.142785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.142758    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"]
Oct 31 02:00:01.147455 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.147423    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"]
Oct 31 02:00:01.616332 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.616305    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerStarted","Data":"0fd22cc040063a7edbc0a45d174f893382b7de2e95f4ff44739ee19f4a684c45"}
Oct 31 02:00:01.616770 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.616364    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerStarted","Data":"99300b0b87efb92efa45f985e5ab52969d233215fa1812631a4c40bf6068fa07"}
Oct 31 02:00:01.617776 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.617753    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerStarted","Data":"c8eced6aef0c482758a0dcaaa946932aef4b0ba1c2226754188481b0a6645d3f"}
Oct 31 02:00:01.619324 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.619297    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerStarted","Data":"eb886b53bb0014c198cf948fdf85d5f500a73cebb5b1e9bc39fce50620716486"}
Oct 31 02:00:01.619426 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.619348    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerStarted","Data":"516cc95e8ed5a03a23c11aecf9a8d785d0ff4b2cdf6522419f6afa326610b4b6"}
Oct 31 02:00:01.620965 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.620940    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerStarted","Data":"289b3aab25711234d9a48a7298b9302d4e9596b772967da1d3f70059af5948b5"}
Oct 31 02:00:01.621060 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.620982    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerStarted","Data":"2d3870df90641f15d9ea86acaf5349f115a5d801f844192d4323a9001666e5fa"}
Oct 31 02:00:01.622416 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.622398    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerStarted","Data":"445a922190b2441e1a3fee56bd7ec710837e1e3e66e6816ff4c867c4adeb1ce7"}
Oct 31 02:00:01.622509 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.622439    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerStarted","Data":"520400e27c1b5abc0a386f4da3ab7662f9268c00fa51b87e03ba327edc1e1614"}
Oct 31 02:00:01.623732 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.623709    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerStarted","Data":"f6f9d1063dd990c705e97e2c8817ea6f273c1ba681cbd0db6093d42557d10e7c"}
Oct 31 02:00:01.625185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.625161    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerStarted","Data":"308eb03073ae52eb00819716a0ce6243ed14d295f53b991d3182cd542850307d"}
Oct 31 02:00:01.625281 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.625206    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerStarted","Data":"975293acf943a2c022d9520021f0f8493b292bca9e7b084e6eaeee26f94de5dc"}
Oct 31 02:00:01.626476 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.626454    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerStarted","Data":"bf1d83a3bf6ac98543a33dcf07425de51ac390099a214bba2badf2c27a1c8702"}
Oct 31 02:00:01.627567 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.627549    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerStarted","Data":"36e226a5ba8187a43b1be2935b01512ec3c4fea5ec7706896f7ff5af443b3099"}
Oct 31 02:00:01.729401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.729355    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" podStartSLOduration=1.7293428149999999 podStartE2EDuration="1.729342815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.727415243 +0000 UTC m=+3323.028371831" watchObservedRunningTime="2026-10-31 02:00:01.729342815 +0000 UTC m=+3323.030299433"
Oct 31 02:00:01.842672 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.842584    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" podStartSLOduration=1.8425710309999999 podStartE2EDuration="1.842571031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.839280084 +0000 UTC m=+3323.140236680" watchObservedRunningTime="2026-10-31 02:00:01.842571031 +0000 UTC m=+3323.143527608"
Oct 31 02:00:01.955119 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:01.955069    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" podStartSLOduration=1.955056498 podStartE2EDuration="1.955056498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.951772612 +0000 UTC m=+3323.252729190" watchObservedRunningTime="2026-10-31 02:00:01.955056498 +0000 UTC m=+3323.256013074"
Oct 31 02:00:02.067913 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.067855    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" podStartSLOduration=2.067841516 podStartE2EDuration="2.067841516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.064438185 +0000 UTC m=+3323.365394763" watchObservedRunningTime="2026-10-31 02:00:02.067841516 +0000 UTC m=+3323.368798092"
Oct 31 02:00:02.191780 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.191727    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" podStartSLOduration=2.191712371 podStartE2EDuration="2.191712371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.190392718 +0000 UTC m=+3323.491349295" watchObservedRunningTime="2026-10-31 02:00:02.191712371 +0000 UTC m=+3323.492668948"
Oct 31 02:00:02.636481 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.636401    5419 generic.go:356] "Generic (PLEG): container finished" podID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerID="6ac1c446bdf917fe9ffc860174a2b4788ab92f208d86d011b89bcce9d385d10c" exitCode=60
Oct 31 02:00:02.636877 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.636489    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerDied","Data":"6ac1c446bdf917fe9ffc860174a2b4788ab92f208d86d011b89bcce9d385d10c"}
Oct 31 02:00:02.636877 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.636763    5419 scope.go:122] "RemoveContainer" containerID="6ac1c446bdf917fe9ffc860174a2b4788ab92f208d86d011b89bcce9d385d10c"
Oct 31 02:00:02.638231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.638207    5419 generic.go:356] "Generic (PLEG): container finished" podID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerID="acbb3ec3b62f700e5f1b814393b8c4d0d7b8e2c0f73b9eb198c62ec1f1532301" exitCode=60
Oct 31 02:00:02.638366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.638292    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerDied","Data":"acbb3ec3b62f700e5f1b814393b8c4d0d7b8e2c0f73b9eb198c62ec1f1532301"}
Oct 31 02:00:02.638541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.638524    5419 scope.go:122] "RemoveContainer" containerID="acbb3ec3b62f700e5f1b814393b8c4d0d7b8e2c0f73b9eb198c62ec1f1532301"
Oct 31 02:00:02.640464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.640442    5419 generic.go:356] "Generic (PLEG): container finished" podID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerID="0fd22cc040063a7edbc0a45d174f893382b7de2e95f4ff44739ee19f4a684c45" exitCode=60
Oct 31 02:00:02.640570 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.640524    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerDied","Data":"0fd22cc040063a7edbc0a45d174f893382b7de2e95f4ff44739ee19f4a684c45"}
Oct 31 02:00:02.640745 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.640726    5419 scope.go:122] "RemoveContainer" containerID="0fd22cc040063a7edbc0a45d174f893382b7de2e95f4ff44739ee19f4a684c45"
Oct 31 02:00:02.642224 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.642203    5419 generic.go:356] "Generic (PLEG): container finished" podID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerID="761c7117fb4709135f6b9f31b1b2678bb204fc947d9beccd46a8bcefcff477eb" exitCode=60
Oct 31 02:00:02.642338 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.642289    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerDied","Data":"761c7117fb4709135f6b9f31b1b2678bb204fc947d9beccd46a8bcefcff477eb"}
Oct 31 02:00:02.642481 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.642469    5419 scope.go:122] "RemoveContainer" containerID="761c7117fb4709135f6b9f31b1b2678bb204fc947d9beccd46a8bcefcff477eb"
Oct 31 02:00:02.643875 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.643854    5419 generic.go:356] "Generic (PLEG): container finished" podID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerID="eb886b53bb0014c198cf948fdf85d5f500a73cebb5b1e9bc39fce50620716486" exitCode=60
Oct 31 02:00:02.643981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.643936    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerDied","Data":"eb886b53bb0014c198cf948fdf85d5f500a73cebb5b1e9bc39fce50620716486"}
Oct 31 02:00:02.644187 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.644172    5419 scope.go:122] "RemoveContainer" containerID="eb886b53bb0014c198cf948fdf85d5f500a73cebb5b1e9bc39fce50620716486"
Oct 31 02:00:02.645545 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.645523    5419 generic.go:356] "Generic (PLEG): container finished" podID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerID="289b3aab25711234d9a48a7298b9302d4e9596b772967da1d3f70059af5948b5" exitCode=60
Oct 31 02:00:02.645630 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.645613    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerDied","Data":"289b3aab25711234d9a48a7298b9302d4e9596b772967da1d3f70059af5948b5"}
Oct 31 02:00:02.645840 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.645822    5419 scope.go:122] "RemoveContainer" containerID="289b3aab25711234d9a48a7298b9302d4e9596b772967da1d3f70059af5948b5"
Oct 31 02:00:02.647189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.647170    5419 generic.go:356] "Generic (PLEG): container finished" podID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerID="445a922190b2441e1a3fee56bd7ec710837e1e3e66e6816ff4c867c4adeb1ce7" exitCode=60
Oct 31 02:00:02.647284 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.647248    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerDied","Data":"445a922190b2441e1a3fee56bd7ec710837e1e3e66e6816ff4c867c4adeb1ce7"}
Oct 31 02:00:02.647440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.647419    5419 scope.go:122] "RemoveContainer" containerID="445a922190b2441e1a3fee56bd7ec710837e1e3e66e6816ff4c867c4adeb1ce7"
Oct 31 02:00:02.649875 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.649851    5419 generic.go:356] "Generic (PLEG): container finished" podID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerID="e0210b073cbde878ce0a654de6e4146ad8975982ff60fceef755ebebec6049ad" exitCode=60
Oct 31 02:00:02.649971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.649956    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerDied","Data":"e0210b073cbde878ce0a654de6e4146ad8975982ff60fceef755ebebec6049ad"}
Oct 31 02:00:02.650093 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.650065    5419 scope.go:122] "RemoveContainer" containerID="e0210b073cbde878ce0a654de6e4146ad8975982ff60fceef755ebebec6049ad"
Oct 31 02:00:02.651714 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.651694    5419 generic.go:356] "Generic (PLEG): container finished" podID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerID="308eb03073ae52eb00819716a0ce6243ed14d295f53b991d3182cd542850307d" exitCode=60
Oct 31 02:00:02.651816 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.651756    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerDied","Data":"308eb03073ae52eb00819716a0ce6243ed14d295f53b991d3182cd542850307d"}
Oct 31 02:00:02.651908 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:02.651894    5419 scope.go:122] "RemoveContainer" containerID="308eb03073ae52eb00819716a0ce6243ed14d295f53b991d3182cd542850307d"
Oct 31 02:00:03.204384 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.204358    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:00:03.663663 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.663627    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerStarted","Data":"c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531"}
Oct 31 02:00:03.665938 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.665909    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerStarted","Data":"c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898"}
Oct 31 02:00:03.668500 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.668474    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerStarted","Data":"023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d"}
Oct 31 02:00:03.670773 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.670747    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerStarted","Data":"91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e"}
Oct 31 02:00:03.673305 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.673274    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerStarted","Data":"8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4"}
Oct 31 02:00:03.683520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.675737    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerStarted","Data":"f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d"}
Oct 31 02:00:03.683520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.683062    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerStarted","Data":"c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440"}
Oct 31 02:00:03.685534 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.685508    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerStarted","Data":"3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3"}
Oct 31 02:00:03.688131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.688108    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerStarted","Data":"1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92"}
Oct 31 02:00:03.914100 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:03.913935    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" podStartSLOduration=3.913915637 podStartE2EDuration="3.913915637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:03.905860385 +0000 UTC m=+3325.206816960" watchObservedRunningTime="2026-10-31 02:00:03.913915637 +0000 UTC m=+3325.214872217"
Oct 31 02:00:04.256343 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.256296    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" podStartSLOduration=4.256279508 podStartE2EDuration="4.256279508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.253325486 +0000 UTC m=+3325.554282081" watchObservedRunningTime="2026-10-31 02:00:04.256279508 +0000 UTC m=+3325.557236101"
Oct 31 02:00:04.368691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.368630    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" podStartSLOduration=4.368617708 podStartE2EDuration="4.368617708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.365559541 +0000 UTC m=+3325.666516120" watchObservedRunningTime="2026-10-31 02:00:04.368617708 +0000 UTC m=+3325.669574284"
Oct 31 02:00:04.616178 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.616102    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" podStartSLOduration=4.616087298 podStartE2EDuration="4.616087298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:04.614249207 +0000 UTC m=+3325.915205784" watchObservedRunningTime="2026-10-31 02:00:04.616087298 +0000 UTC m=+3325.917043874"
Oct 31 02:00:04.699878 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.699839    5419 generic.go:356] "Generic (PLEG): container finished" podID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerID="1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92" exitCode=60
Oct 31 02:00:04.700389 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.699897    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerDied","Data":"1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92"}
Oct 31 02:00:04.700389 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.699969    5419 scope.go:122] "RemoveContainer" containerID="eb886b53bb0014c198cf948fdf85d5f500a73cebb5b1e9bc39fce50620716486"
Oct 31 02:00:04.700389 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.700290    5419 scope.go:122] "RemoveContainer" containerID="1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92"
Oct 31 02:00:04.700687 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.700667    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp_odh-ai-gateway-infra(13f5da2d-3a9a-4e1e-bd12-e0847f2b638d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d"
Oct 31 02:00:04.702278 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.702253    5419 generic.go:356] "Generic (PLEG): container finished" podID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerID="c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531" exitCode=60
Oct 31 02:00:04.702388 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.702332    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerDied","Data":"c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531"}
Oct 31 02:00:04.702623 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.702605    5419 scope.go:122] "RemoveContainer" containerID="c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531"
Oct 31 02:00:04.702956 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.702936    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5_odh-ai-gateway-infra(4ad6fe6e-82e3-483c-b421-36c3802957bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc"
Oct 31 02:00:04.704403 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.704382    5419 generic.go:356] "Generic (PLEG): container finished" podID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerID="c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898" exitCode=60
Oct 31 02:00:04.704509 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.704446    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerDied","Data":"c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898"}
Oct 31 02:00:04.704788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.704772    5419 scope.go:122] "RemoveContainer" containerID="c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898"
Oct 31 02:00:04.705274 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.705249    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc_odh-ai-gateway-infra(f69de7d1-e1d7-4d5d-89c9-606556d6f7ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec"
Oct 31 02:00:04.706615 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.706594    5419 generic.go:356] "Generic (PLEG): container finished" podID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerID="023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d" exitCode=60
Oct 31 02:00:04.706707 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.706676    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerDied","Data":"023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d"}
Oct 31 02:00:04.706903 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.706889    5419 scope.go:122] "RemoveContainer" containerID="023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d"
Oct 31 02:00:04.707222 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.707120    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8_odh-ai-gateway-infra(73be4d39-b90a-4ce9-b13d-03f5320e1ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8"
Oct 31 02:00:04.708923 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.708891    5419 generic.go:356] "Generic (PLEG): container finished" podID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerID="91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e" exitCode=60
Oct 31 02:00:04.709032 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.709000    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerDied","Data":"91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e"}
Oct 31 02:00:04.709177 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.709162    5419 scope.go:122] "RemoveContainer" containerID="91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e"
Oct 31 02:00:04.709393 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.709374    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm_odh-ai-gateway-infra(a49161a2-5532-4906-92d5-33d4fd8cbc86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86"
Oct 31 02:00:04.711016 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.710992    5419 generic.go:356] "Generic (PLEG): container finished" podID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerID="8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4" exitCode=60
Oct 31 02:00:04.711257 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.711082    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerDied","Data":"8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4"}
Oct 31 02:00:04.711795 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.711779    5419 scope.go:122] "RemoveContainer" containerID="8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4"
Oct 31 02:00:04.712027 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.712005    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx_odh-ai-gateway-infra(56b0de57-5fad-4a3f-9e06-d7899b82251f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f"
Oct 31 02:00:04.713395 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.713374    5419 generic.go:356] "Generic (PLEG): container finished" podID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerID="f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d" exitCode=60
Oct 31 02:00:04.713496 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.713452    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerDied","Data":"f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d"}
Oct 31 02:00:04.713654 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.713639    5419 scope.go:122] "RemoveContainer" containerID="f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d"
Oct 31 02:00:04.713883 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.713862    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm_odh-ai-gateway-infra(14b838a8-fe5b-4e04-a75f-9a8db2693104)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104"
Oct 31 02:00:04.715446 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.715428    5419 generic.go:356] "Generic (PLEG): container finished" podID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerID="c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440" exitCode=60
Oct 31 02:00:04.715561 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.715487    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerDied","Data":"c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440"}
Oct 31 02:00:04.715734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.715718    5419 scope.go:122] "RemoveContainer" containerID="c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440"
Oct 31 02:00:04.715967 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.715943    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-lw8tv_odh-ai-gateway-infra(3d4b1214-a89c-4107-b7d0-971463d0c16f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f"
Oct 31 02:00:04.717222 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.717204    5419 scope.go:122] "RemoveContainer" containerID="289b3aab25711234d9a48a7298b9302d4e9596b772967da1d3f70059af5948b5"
Oct 31 02:00:04.717493 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.717472    5419 generic.go:356] "Generic (PLEG): container finished" podID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerID="3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3" exitCode=60
Oct 31 02:00:04.717605 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.717511    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerDied","Data":"3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3"}
Oct 31 02:00:04.717900 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.717878    5419 scope.go:122] "RemoveContainer" containerID="3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3"
Oct 31 02:00:04.718467 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:04.718439    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc_odh-ai-gateway-infra(d0b49020-b109-4499-98e2-d39675bfe1dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc"
Oct 31 02:00:04.732672 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.732644    5419 scope.go:122] "RemoveContainer" containerID="445a922190b2441e1a3fee56bd7ec710837e1e3e66e6816ff4c867c4adeb1ce7"
Oct 31 02:00:04.746126 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.746102    5419 scope.go:122] "RemoveContainer" containerID="e0210b073cbde878ce0a654de6e4146ad8975982ff60fceef755ebebec6049ad"
Oct 31 02:00:04.759732 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.759712    5419 scope.go:122] "RemoveContainer" containerID="308eb03073ae52eb00819716a0ce6243ed14d295f53b991d3182cd542850307d"
Oct 31 02:00:04.778531 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.775845    5419 scope.go:122] "RemoveContainer" containerID="6ac1c446bdf917fe9ffc860174a2b4788ab92f208d86d011b89bcce9d385d10c"
Oct 31 02:00:04.792671 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.792646    5419 scope.go:122] "RemoveContainer" containerID="acbb3ec3b62f700e5f1b814393b8c4d0d7b8e2c0f73b9eb198c62ec1f1532301"
Oct 31 02:00:04.809168 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.809127    5419 scope.go:122] "RemoveContainer" containerID="0fd22cc040063a7edbc0a45d174f893382b7de2e95f4ff44739ee19f4a684c45"
Oct 31 02:00:04.822721 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:04.822700    5419 scope.go:122] "RemoveContainer" containerID="761c7117fb4709135f6b9f31b1b2678bb204fc947d9beccd46a8bcefcff477eb"
Oct 31 02:00:05.726654 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.726623    5419 scope.go:122] "RemoveContainer" containerID="1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92"
Oct 31 02:00:05.727101 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.727002    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp_odh-ai-gateway-infra(13f5da2d-3a9a-4e1e-bd12-e0847f2b638d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d"
Oct 31 02:00:05.728407 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.728383    5419 scope.go:122] "RemoveContainer" containerID="c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531"
Oct 31 02:00:05.728723 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.728697    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5_odh-ai-gateway-infra(4ad6fe6e-82e3-483c-b421-36c3802957bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc"
Oct 31 02:00:05.729992 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.729966    5419 scope.go:122] "RemoveContainer" containerID="c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898"
Oct 31 02:00:05.730311 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.730292    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc_odh-ai-gateway-infra(f69de7d1-e1d7-4d5d-89c9-606556d6f7ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec"
Oct 31 02:00:05.731609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.731590    5419 scope.go:122] "RemoveContainer" containerID="023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d"
Oct 31 02:00:05.731884 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.731862    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8_odh-ai-gateway-infra(73be4d39-b90a-4ce9-b13d-03f5320e1ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8"
Oct 31 02:00:05.733194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.733176    5419 scope.go:122] "RemoveContainer" containerID="91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e"
Oct 31 02:00:05.733434 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.733416    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm_odh-ai-gateway-infra(a49161a2-5532-4906-92d5-33d4fd8cbc86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86"
Oct 31 02:00:05.734748 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.734730    5419 scope.go:122] "RemoveContainer" containerID="8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4"
Oct 31 02:00:05.734990 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.734967    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx_odh-ai-gateway-infra(56b0de57-5fad-4a3f-9e06-d7899b82251f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f"
Oct 31 02:00:05.736476 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.736456    5419 scope.go:122] "RemoveContainer" containerID="f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d"
Oct 31 02:00:05.736714 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.736692    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm_odh-ai-gateway-infra(14b838a8-fe5b-4e04-a75f-9a8db2693104)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104"
Oct 31 02:00:05.737959 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.737942    5419 scope.go:122] "RemoveContainer" containerID="c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440"
Oct 31 02:00:05.738198 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.738178    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-lw8tv_odh-ai-gateway-infra(3d4b1214-a89c-4107-b7d0-971463d0c16f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f"
Oct 31 02:00:05.739506 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:05.739491    5419 scope.go:122] "RemoveContainer" containerID="3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3"
Oct 31 02:00:05.739694 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:05.739678    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc_odh-ai-gateway-infra(d0b49020-b109-4499-98e2-d39675bfe1dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc"
Oct 31 02:00:11.325510 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.325476    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"]
Oct 31 02:00:11.329768 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.329748    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454011    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454054    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454341 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454185    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ktf\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454341 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454219    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454341 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454331    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454474 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454368    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454527 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454508    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454580    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.454679 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.454662    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.493444 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.493116    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8f6b5b-openshift-default-dockercfg-6t9gt\""
Oct 31 02:00:11.498612 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.497893    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"]
Oct 31 02:00:11.498612 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.498217    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy" containerID="cri-o://8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288" gracePeriod=30
Oct 31 02:00:11.506171 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.503694    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"]
Oct 31 02:00:11.506171 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.503957    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy" containerID="cri-o://db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037" gracePeriod=30
Oct 31 02:00:11.535986 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.535951    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"]
Oct 31 02:00:11.563617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563316    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ktf\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.563617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563365    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.563617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563457    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.563617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563503    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.563617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563606    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.564390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563687    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.564390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563748    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.564390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563844    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.564390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.563887    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.564595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.564417    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.564996 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.564971    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.567198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.566495    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.567198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.566532    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.567198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.566761    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.585402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.567802    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.602185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.598450    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.623069 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.623034    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.623731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.623714    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ktf\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf\") pod \"e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") " pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:11.643559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:11.643529    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:12.074298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:12.074268    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"]
Oct 31 02:00:12.075683 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:12.075662    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db42ddf_b07d_4263_bc7a_fd9ec331f8a0.slice/crio-3bec60159c4bef164de5622b19ee90ccece5ca5b0136febdc4da6c0f2f1ab636 WatchSource:0}: Error finding container 3bec60159c4bef164de5622b19ee90ccece5ca5b0136febdc4da6c0f2f1ab636: Status 404 returned error can't find the container with id 3bec60159c4bef164de5622b19ee90ccece5ca5b0136febdc4da6c0f2f1ab636
Oct 31 02:00:12.078262 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:12.078229    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 02:00:12.078345 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:12.078305    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 02:00:12.078345 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:12.078338    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 02:00:12.845935 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:12.845892    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" event={"ID":"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0","Type":"ContainerStarted","Data":"d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503"}
Oct 31 02:00:12.846618 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:12.846593    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" event={"ID":"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0","Type":"ContainerStarted","Data":"3bec60159c4bef164de5622b19ee90ccece5ca5b0136febdc4da6c0f2f1ab636"}
Oct 31 02:00:13.049384 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:13.049335    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podStartSLOduration=2.049321804 podStartE2EDuration="2.049321804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:13.049044045 +0000 UTC m=+3334.350000622" watchObservedRunningTime="2026-10-31 02:00:13.049321804 +0000 UTC m=+3334.350278380"
Oct 31 02:00:13.643818 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:13.643791    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:13.645025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:13.645004    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:13.645075 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:13.645055    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:13.830342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:13.830314    5419 scope.go:122] "RemoveContainer" containerID="c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440"
Oct 31 02:00:14.607098 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:14.607058    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:00:14.644570 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:14.644532    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:14.644736 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:14.644589    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:14.830336 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:14.830305    5419 scope.go:122] "RemoveContainer" containerID="c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898"
Oct 31 02:00:14.830518 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:14.830423    5419 scope.go:122] "RemoveContainer" containerID="1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92"
Oct 31 02:00:14.830570 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:14.830558    5419 scope.go:122] "RemoveContainer" containerID="3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3"
Oct 31 02:00:14.859957 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:14.859930    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerStarted","Data":"cab696e0ff7504aed6110f76274a1f089242c7d12609206e3c7b38a66710bdf3"}
Oct 31 02:00:15.644346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.644311    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:15.644743 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.644364    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:15.831022 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.830985    5419 scope.go:122] "RemoveContainer" containerID="91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e"
Oct 31 02:00:15.831217 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.831094    5419 scope.go:122] "RemoveContainer" containerID="023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d"
Oct 31 02:00:15.870219 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.870181    5419 generic.go:356] "Generic (PLEG): container finished" podID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerID="cab696e0ff7504aed6110f76274a1f089242c7d12609206e3c7b38a66710bdf3" exitCode=60
Oct 31 02:00:15.870332 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.870252    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerDied","Data":"cab696e0ff7504aed6110f76274a1f089242c7d12609206e3c7b38a66710bdf3"}
Oct 31 02:00:15.870332 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.870319    5419 scope.go:122] "RemoveContainer" containerID="c25064b48433ad75810a541c12870d80eea3466a03f1d2f0010c52f9777f1440"
Oct 31 02:00:15.870699 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.870682    5419 scope.go:122] "RemoveContainer" containerID="cab696e0ff7504aed6110f76274a1f089242c7d12609206e3c7b38a66710bdf3"
Oct 31 02:00:15.871000 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:15.870979    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-lw8tv_odh-ai-gateway-infra(3d4b1214-a89c-4107-b7d0-971463d0c16f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f"
Oct 31 02:00:15.872647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.872625    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerStarted","Data":"fe7953e4d041be4950d6136b76a56095ec3e5501d566a96eee1541a4496cbc0c"}
Oct 31 02:00:15.901204 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.897588    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerStarted","Data":"e5d65a02b429fa528276e101be2cca30fce64861380e5d2efc5f776e1be88868"}
Oct 31 02:00:15.914176 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:15.909987    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerStarted","Data":"b13b5f43ef57e0a8062ce8e948ac2dbf40a05e465e5daba7e433baea4fee212b"}
Oct 31 02:00:16.012447 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.012416    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"]
Oct 31 02:00:16.645766 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.645034    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:16.645766 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.645096    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:16.821418 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.821392    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 02:00:16.843655 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.843625    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.843825 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.843674    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.843825 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.843749    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.843825 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.843777    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.843851    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.843949    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844006    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844069    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844185    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844238    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket\") pod \"ba157891-0965-474c-af24-d4deabbfc812\" (UID: \"ba157891-0965-474c-af24-d4deabbfc812\") "
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844374    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844581    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844631    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.844784    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.845049    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.845070    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.845085    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.845100    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.845677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.845113    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba157891-0965-474c-af24-d4deabbfc812-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.846366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.846265    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:16.847210 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.847191    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:16.847296 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.847233    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "kube-api-access-t55w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:16.847296 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.847238    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy" pod "ba157891-0965-474c-af24-d4deabbfc812" (UID: "ba157891-0965-474c-af24-d4deabbfc812"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.860542 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.860520    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 02:00:16.922227 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.922181    5419 generic.go:356] "Generic (PLEG): container finished" podID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerID="b13b5f43ef57e0a8062ce8e948ac2dbf40a05e465e5daba7e433baea4fee212b" exitCode=60
Oct 31 02:00:16.922424 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.922236    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerDied","Data":"b13b5f43ef57e0a8062ce8e948ac2dbf40a05e465e5daba7e433baea4fee212b"}
Oct 31 02:00:16.922424 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.922310    5419 scope.go:122] "RemoveContainer" containerID="c7660380a93fb0eda0228928612acfcc1c216b6a68a55d9025c46f4f60ca1898"
Oct 31 02:00:16.922729 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.922714    5419 scope.go:122] "RemoveContainer" containerID="b13b5f43ef57e0a8062ce8e948ac2dbf40a05e465e5daba7e433baea4fee212b"
Oct 31 02:00:16.923100 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:16.923076    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc_odh-ai-gateway-infra(f69de7d1-e1d7-4d5d-89c9-606556d6f7ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec"
Oct 31 02:00:16.924663 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.924639    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerStarted","Data":"6fdc432b982a7a81579ad1c956ffa2999ee28a2a60ebae0f85b283830698911e"}
Oct 31 02:00:16.927528 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.926845    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerStarted","Data":"88251abb4eb03fc8e8844c265ad448b3e3eb32ffa954fca54604e7fd1be7c78e"}
Oct 31 02:00:16.929251 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.929227    5419 generic.go:356] "Generic (PLEG): container finished" podID="ba157891-0965-474c-af24-d4deabbfc812" containerID="8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288" exitCode=0
Oct 31 02:00:16.929430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.929409    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"
Oct 31 02:00:16.929965 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.929938    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" event={"ID":"ba157891-0965-474c-af24-d4deabbfc812","Type":"ContainerDied","Data":"8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288"}
Oct 31 02:00:16.930076 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.929991    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf" event={"ID":"ba157891-0965-474c-af24-d4deabbfc812","Type":"ContainerDied","Data":"a7be6496827521292ef54229ce47a32f6719cba617d048145013695a08cd2cc2"}
Oct 31 02:00:16.933172 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.932484    5419 generic.go:356] "Generic (PLEG): container finished" podID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerID="db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037" exitCode=0
Oct 31 02:00:16.933172 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.932510    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" event={"ID":"51b31c94-8461-4c4d-88cd-9bdf60eeac63","Type":"ContainerDied","Data":"db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037"}
Oct 31 02:00:16.933172 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.932539    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"
Oct 31 02:00:16.933172 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.932550    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr" event={"ID":"51b31c94-8461-4c4d-88cd-9bdf60eeac63","Type":"ContainerDied","Data":"cc01600c76eed00724c9460d6e3cb6cdf608563c0ec11a9ef69acbc6f46ee316"}
Oct 31 02:00:16.936826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.936804    5419 generic.go:356] "Generic (PLEG): container finished" podID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerID="fe7953e4d041be4950d6136b76a56095ec3e5501d566a96eee1541a4496cbc0c" exitCode=60
Oct 31 02:00:16.937252 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.936877    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerDied","Data":"fe7953e4d041be4950d6136b76a56095ec3e5501d566a96eee1541a4496cbc0c"}
Oct 31 02:00:16.937552 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.937266    5419 scope.go:122] "RemoveContainer" containerID="fe7953e4d041be4950d6136b76a56095ec3e5501d566a96eee1541a4496cbc0c"
Oct 31 02:00:16.937646 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:16.937611    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc_odh-ai-gateway-infra(d0b49020-b109-4499-98e2-d39675bfe1dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc"
Oct 31 02:00:16.939300 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.939267    5419 scope.go:122] "RemoveContainer" containerID="8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288"
Oct 31 02:00:16.940677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.940636    5419 generic.go:356] "Generic (PLEG): container finished" podID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerID="e5d65a02b429fa528276e101be2cca30fce64861380e5d2efc5f776e1be88868" exitCode=60
Oct 31 02:00:16.940776 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.940693    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerDied","Data":"e5d65a02b429fa528276e101be2cca30fce64861380e5d2efc5f776e1be88868"}
Oct 31 02:00:16.941077 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.941058    5419 scope.go:122] "RemoveContainer" containerID="e5d65a02b429fa528276e101be2cca30fce64861380e5d2efc5f776e1be88868"
Oct 31 02:00:16.941892 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:16.941850    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp_odh-ai-gateway-infra(13f5da2d-3a9a-4e1e-bd12-e0847f2b638d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d"
Oct 31 02:00:16.951874 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.951847    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.952191 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952169    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.952385 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952367    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.952558 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952544    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.952693 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952680    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.952818 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952805    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.953067 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952373    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.953067 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.952515    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.953236 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.953087    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.953701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.953337    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.953701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.953431    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:16.953701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.953435    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.953701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.953549    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket\") pod \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\" (UID: \"51b31c94-8461-4c4d-88cd-9bdf60eeac63\") "
Oct 31 02:00:16.954066 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.953890    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954559    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954751    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954774    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954788    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba157891-0965-474c-af24-d4deabbfc812-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954811    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954825    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba157891-0965-474c-af24-d4deabbfc812-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954838    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.954856 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954853    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55w2\" (UniqueName: \"kubernetes.io/projected/ba157891-0965-474c-af24-d4deabbfc812-kube-api-access-t55w2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.955340 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.954868    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:16.955651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.955619    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:16.955793 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.955695    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:16.957602 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.957580    5419 scope.go:122] "RemoveContainer" containerID="8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288"
Oct 31 02:00:16.957947 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:16.957925    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288\": container with ID starting with 8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288 not found: ID does not exist" containerID="8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288"
Oct 31 02:00:16.958027 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.957955    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288"} err="failed to get container status \"8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288\": rpc error: code = NotFound desc = could not find container \"8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288\": container with ID starting with 8c4765a5d3c20a870a630b0eafd16cc35eddaaa83fa84c5c78b1e6255a992288 not found: ID does not exist"
Oct 31 02:00:16.958027 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.957974    5419 scope.go:122] "RemoveContainer" containerID="db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037"
Oct 31 02:00:16.958426 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.958271    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:16.966299 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:16.965292    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx" pod "51b31c94-8461-4c4d-88cd-9bdf60eeac63" (UID: "51b31c94-8461-4c4d-88cd-9bdf60eeac63"). InnerVolumeSpecName "kube-api-access-tp6nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:17.095970 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.093924    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:17.095970 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.093955    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:17.095970 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.093973    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b31c94-8461-4c4d-88cd-9bdf60eeac63-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:17.095970 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.093988    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp6nx\" (UniqueName: \"kubernetes.io/projected/51b31c94-8461-4c4d-88cd-9bdf60eeac63-kube-api-access-tp6nx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:17.144651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.144462    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"]
Oct 31 02:00:17.144763 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.144684    5419 scope.go:122] "RemoveContainer" containerID="db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037"
Oct 31 02:00:17.145078 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:17.145046    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037\": container with ID starting with db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037 not found: ID does not exist" containerID="db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037"
Oct 31 02:00:17.145194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.145088    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037"} err="failed to get container status \"db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037\": rpc error: code = NotFound desc = could not find container \"db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037\": container with ID starting with db82dd71d2f7cbbe410725b532f12059ccaf06c8dcd42249f746e0525c2de037 not found: ID does not exist"
Oct 31 02:00:17.145194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.145114    5419 scope.go:122] "RemoveContainer" containerID="3e9ff944fab6545db00c5cd19c11cdd6f538abf11b35811a51807d81b03581f3"
Oct 31 02:00:17.169987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.169964    5419 scope.go:122] "RemoveContainer" containerID="1eee4a2b3fa43fdce948cf44fb20abdeb620cd910878b2419dee6b0b8cf18c92"
Oct 31 02:00:17.219670 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.219638    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:17.250395 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.250368    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"]
Oct 31 02:00:17.310154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.310055    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl\" (UniqueName: \"kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl\") pod \"3d4b1214-a89c-4107-b7d0-971463d0c16f\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") "
Oct 31 02:00:17.310533 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.310516    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca\") pod \"3d4b1214-a89c-4107-b7d0-971463d0c16f\" (UID: \"3d4b1214-a89c-4107-b7d0-971463d0c16f\") "
Oct 31 02:00:17.310908 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.310888    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca" pod "3d4b1214-a89c-4107-b7d0-971463d0c16f" (UID: "3d4b1214-a89c-4107-b7d0-971463d0c16f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:17.312977 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.312955    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl" pod "3d4b1214-a89c-4107-b7d0-971463d0c16f" (UID: "3d4b1214-a89c-4107-b7d0-971463d0c16f"). InnerVolumeSpecName "kube-api-access-85tpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:17.366852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.366825    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"]
Oct 31 02:00:17.412112 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.412079    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85tpl\" (UniqueName: \"kubernetes.io/projected/3d4b1214-a89c-4107-b7d0-971463d0c16f-kube-api-access-85tpl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:17.412112 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.412106    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4b1214-a89c-4107-b7d0-971463d0c16f-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:17.644013 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.643929    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:17.644013 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.643983    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:17.830405 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.830377    5419 scope.go:122] "RemoveContainer" containerID="8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4"
Oct 31 02:00:17.830788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.830477    5419 scope.go:122] "RemoveContainer" containerID="c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531"
Oct 31 02:00:17.830788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.830523    5419 scope.go:122] "RemoveContainer" containerID="f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d"
Oct 31 02:00:17.906248 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.906214    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"]
Oct 31 02:00:17.910064 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.910036    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-snlqf"]
Oct 31 02:00:17.955292 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.955268    5419 generic.go:356] "Generic (PLEG): container finished" podID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerID="6fdc432b982a7a81579ad1c956ffa2999ee28a2a60ebae0f85b283830698911e" exitCode=60
Oct 31 02:00:17.955397 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.955340    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerDied","Data":"6fdc432b982a7a81579ad1c956ffa2999ee28a2a60ebae0f85b283830698911e"}
Oct 31 02:00:17.955482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.955407    5419 scope.go:122] "RemoveContainer" containerID="023de5f177c2dc821b332ce137b42362e7e731d87818acfb8cc7b582b9c3224d"
Oct 31 02:00:17.955792 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.955776    5419 scope.go:122] "RemoveContainer" containerID="6fdc432b982a7a81579ad1c956ffa2999ee28a2a60ebae0f85b283830698911e"
Oct 31 02:00:17.956214 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:17.956121    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8_odh-ai-gateway-infra(73be4d39-b90a-4ce9-b13d-03f5320e1ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8"
Oct 31 02:00:17.957764 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.957744    5419 generic.go:356] "Generic (PLEG): container finished" podID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerID="88251abb4eb03fc8e8844c265ad448b3e3eb32ffa954fca54604e7fd1be7c78e" exitCode=60
Oct 31 02:00:17.957867 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.957817    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerDied","Data":"88251abb4eb03fc8e8844c265ad448b3e3eb32ffa954fca54604e7fd1be7c78e"}
Oct 31 02:00:17.958254 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.958235    5419 scope.go:122] "RemoveContainer" containerID="88251abb4eb03fc8e8844c265ad448b3e3eb32ffa954fca54604e7fd1be7c78e"
Oct 31 02:00:17.958638 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:17.958590    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm_odh-ai-gateway-infra(a49161a2-5532-4906-92d5-33d4fd8cbc86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86"
Oct 31 02:00:17.962271 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.962257    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"
Oct 31 02:00:17.965495 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.962292    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv" event={"ID":"3d4b1214-a89c-4107-b7d0-971463d0c16f","Type":"ContainerDied","Data":"99300b0b87efb92efa45f985e5ab52969d233215fa1812631a4c40bf6068fa07"}
Oct 31 02:00:17.988731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:17.988404    5419 scope.go:122] "RemoveContainer" containerID="91da2922ba4aaf58314cdfb71faa4bfed5a768f14c55d9c8c9036ae1fa46988e"
Oct 31 02:00:18.017883 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.017854    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"]
Oct 31 02:00:18.021832 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.021812    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-j5ggr"]
Oct 31 02:00:18.033796 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.033255    5419 scope.go:122] "RemoveContainer" containerID="cab696e0ff7504aed6110f76274a1f089242c7d12609206e3c7b38a66710bdf3"
Oct 31 02:00:18.069327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.069299    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"]
Oct 31 02:00:18.270505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.270471    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:18.293972 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.293944    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:18.418850 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.418761    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:18.435939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.432016    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca\") pod \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") "
Oct 31 02:00:18.435939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.432126    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm\" (UniqueName: \"kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm\") pod \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") "
Oct 31 02:00:18.435939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.432182    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns\" (UniqueName: \"kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns\") pod \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\" (UID: \"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec\") "
Oct 31 02:00:18.435939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.432386    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca\") pod \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\" (UID: \"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d\") "
Oct 31 02:00:18.435939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.433254    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca" pod "13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" (UID: "13f5da2d-3a9a-4e1e-bd12-e0847f2b638d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:18.435939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.433557    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca" pod "f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" (UID: "f69de7d1-e1d7-4d5d-89c9-606556d6f7ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:18.436440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.436201    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm" pod "13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" (UID: "13f5da2d-3a9a-4e1e-bd12-e0847f2b638d"). InnerVolumeSpecName "kube-api-access-wbltm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:18.438199 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.438175    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns" pod "f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" (UID: "f69de7d1-e1d7-4d5d-89c9-606556d6f7ec"). InnerVolumeSpecName "kube-api-access-lj9ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:18.533469 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.533433    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv\" (UniqueName: \"kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv\") pod \"d0b49020-b109-4499-98e2-d39675bfe1dc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") "
Oct 31 02:00:18.533678 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.533513    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca\") pod \"d0b49020-b109-4499-98e2-d39675bfe1dc\" (UID: \"d0b49020-b109-4499-98e2-d39675bfe1dc\") "
Oct 31 02:00:18.534122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.534095    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj9ns\" (UniqueName: \"kubernetes.io/projected/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-kube-api-access-lj9ns\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.534122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.534121    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.534122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.534153    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.534401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.534163    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbltm\" (UniqueName: \"kubernetes.io/projected/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d-kube-api-access-wbltm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.534401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.534307    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca" pod "d0b49020-b109-4499-98e2-d39675bfe1dc" (UID: "d0b49020-b109-4499-98e2-d39675bfe1dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:18.536046 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.536018    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv" pod "d0b49020-b109-4499-98e2-d39675bfe1dc" (UID: "d0b49020-b109-4499-98e2-d39675bfe1dc"). InnerVolumeSpecName "kube-api-access-xgcdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:18.674769 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.672322    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"]
Oct 31 02:00:18.674769 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.674117    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:18.674769 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.674190    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:18.676337 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.676306    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcdv\" (UniqueName: \"kubernetes.io/projected/d0b49020-b109-4499-98e2-d39675bfe1dc-kube-api-access-xgcdv\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.676542 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.676524    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0b49020-b109-4499-98e2-d39675bfe1dc-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:18.681675 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.681645    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-lw8tv"]
Oct 31 02:00:18.685527 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.685498    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"]
Oct 31 02:00:18.883197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.875725    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" path="/var/lib/kubelet/pods/3d4b1214-a89c-4107-b7d0-971463d0c16f/volumes"
Oct 31 02:00:18.896485 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.896451    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" path="/var/lib/kubelet/pods/51b31c94-8461-4c4d-88cd-9bdf60eeac63/volumes"
Oct 31 02:00:18.897118 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.897098    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba157891-0965-474c-af24-d4deabbfc812" path="/var/lib/kubelet/pods/ba157891-0965-474c-af24-d4deabbfc812/volumes"
Oct 31 02:00:18.901360 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.901329    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"]
Oct 31 02:00:18.904228 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904197    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:18.904228 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904218    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:18.904228 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904235    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904241    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904253    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904260    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904298    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904304    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904311    5419 cpu_manager.go:414] "containerMap: removing container" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904315    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904321    5419 cpu_manager.go:414] "containerMap: removing container" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904325    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904332    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904337    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904348    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904352    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904360    5419 cpu_manager.go:414] "containerMap: removing container" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904365    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904388    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904392    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904400    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904404    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904422    5419 cpu_manager.go:414] "containerMap: removing container" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.904483 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904427    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904547    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904556    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba157891-0965-474c-af24-d4deabbfc812" containerName="istio-proxy"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904563    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904569    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904574    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904580    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904585    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b31c94-8461-4c4d-88cd-9bdf60eeac63" containerName="istio-proxy"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904592    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904596    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904603    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904610    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904616    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.905681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.904621    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:18.951339 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:18.950562    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.025813 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.007297    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c4be41\""
Oct 31 02:00:19.025813 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.007460    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c4be41-dockercfg-pcf92\""
Oct 31 02:00:19.079274 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.078807    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp" event={"ID":"13f5da2d-3a9a-4e1e-bd12-e0847f2b638d","Type":"ContainerDied","Data":"516cc95e8ed5a03a23c11aecf9a8d785d0ff4b2cdf6522419f6afa326610b4b6"}
Oct 31 02:00:19.079274 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.078902    5419 scope.go:122] "RemoveContainer" containerID="e5d65a02b429fa528276e101be2cca30fce64861380e5d2efc5f776e1be88868"
Oct 31 02:00:19.079274 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.079126    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"
Oct 31 02:00:19.147826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.147764    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.148341 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.148316    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k969w\" (UniqueName: \"kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w\") pod \"payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.172342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.172291    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"]
Oct 31 02:00:19.172623 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.172607    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"]
Oct 31 02:00:19.174278 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.174257    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:19.174417 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.174404    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:19.174522 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.174510    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:19.174613 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.174602    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4b1214-a89c-4107-b7d0-971463d0c16f" containerName="cleanup"
Oct 31 02:00:19.192722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.189634    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" containerName="cleanup"
Oct 31 02:00:19.200247 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.199098    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"
Oct 31 02:00:19.209440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.208576    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerStarted","Data":"f3c453ac7fdd0521469c679663c25b19988ba43edab074091bd19082e272ce7c"}
Oct 31 02:00:19.209440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.208762    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.209616 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.209551    5419 scope.go:122] "RemoveContainer" containerID="f3c453ac7fdd0521469c679663c25b19988ba43edab074091bd19082e272ce7c"
Oct 31 02:00:19.209616 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.208843    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc" event={"ID":"f69de7d1-e1d7-4d5d-89c9-606556d6f7ec","Type":"ContainerDied","Data":"520400e27c1b5abc0a386f4da3ab7662f9268c00fa51b87e03ba327edc1e1614"}
Oct 31 02:00:19.210767 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:19.210289    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5_odh-ai-gateway-infra(4ad6fe6e-82e3-483c-b421-36c3802957bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc"
Oct 31 02:00:19.211380 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.211326    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"]
Oct 31 02:00:19.222920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.222896    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerStarted","Data":"35a9cff36b2c8bd25b921f359ca3b3658912c32df0433b3e38edfa130666a564"}
Oct 31 02:00:19.224011 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.223989    5419 scope.go:122] "RemoveContainer" containerID="35a9cff36b2c8bd25b921f359ca3b3658912c32df0433b3e38edfa130666a564"
Oct 31 02:00:19.224826 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:19.224795    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx_odh-ai-gateway-infra(56b0de57-5fad-4a3f-9e06-d7899b82251f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f"
Oct 31 02:00:19.230979 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.230951    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerStarted","Data":"fed69b724f66b632d3cd8998374e37030d6f7d43e64c5b56432292c3c95098d6"}
Oct 31 02:00:19.231619 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.231599    5419 scope.go:122] "RemoveContainer" containerID="fed69b724f66b632d3cd8998374e37030d6f7d43e64c5b56432292c3c95098d6"
Oct 31 02:00:19.232489 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:19.232466    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm_odh-ai-gateway-infra(14b838a8-fe5b-4e04-a75f-9a8db2693104)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104"
Oct 31 02:00:19.235736 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.235095    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc" event={"ID":"d0b49020-b109-4499-98e2-d39675bfe1dc","Type":"ContainerDied","Data":"c8eced6aef0c482758a0dcaaa946932aef4b0ba1c2226754188481b0a6645d3f"}
Oct 31 02:00:19.235736 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.235238    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"
Oct 31 02:00:19.251779 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.250410    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k969w\" (UniqueName: \"kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w\") pod \"payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.251779 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.250465    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.251779 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.251486    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.260691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.258388    5419 scope.go:122] "RemoveContainer" containerID="b13b5f43ef57e0a8062ce8e948ac2dbf40a05e465e5daba7e433baea4fee212b"
Oct 31 02:00:19.284508 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.284474    5419 scope.go:122] "RemoveContainer" containerID="fe7953e4d041be4950d6136b76a56095ec3e5501d566a96eee1541a4496cbc0c"
Oct 31 02:00:19.336838 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.335054    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k969w\" (UniqueName: \"kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w\") pod \"payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.352170 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.351797    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.352170 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.352065    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl4tn\" (UniqueName: \"kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn\") pod \"payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.410259 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.410231    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:19.453895 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.453327    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl4tn\" (UniqueName: \"kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn\") pod \"payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.455421 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.454583    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.456082 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.456036    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.478933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.478906    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:19.509885 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.509853    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl4tn\" (UniqueName: \"kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn\") pod \"payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.555807 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.555772    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca\") pod \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") "
Oct 31 02:00:19.556010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.555828    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca\") pod \"a49161a2-5532-4906-92d5-33d4fd8cbc86\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") "
Oct 31 02:00:19.556010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.555893    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d\" (UniqueName: \"kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d\") pod \"a49161a2-5532-4906-92d5-33d4fd8cbc86\" (UID: \"a49161a2-5532-4906-92d5-33d4fd8cbc86\") "
Oct 31 02:00:19.556010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.555928    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm\" (UniqueName: \"kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm\") pod \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\" (UID: \"73be4d39-b90a-4ce9-b13d-03f5320e1ea8\") "
Oct 31 02:00:19.556251 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.556227    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca" pod "73be4d39-b90a-4ce9-b13d-03f5320e1ea8" (UID: "73be4d39-b90a-4ce9-b13d-03f5320e1ea8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:19.556324 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.556292    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca" pod "a49161a2-5532-4906-92d5-33d4fd8cbc86" (UID: "a49161a2-5532-4906-92d5-33d4fd8cbc86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:19.556594 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.556573    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.556707 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.556599    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49161a2-5532-4906-92d5-33d4fd8cbc86-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.558722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.558694    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm" pod "73be4d39-b90a-4ce9-b13d-03f5320e1ea8" (UID: "73be4d39-b90a-4ce9-b13d-03f5320e1ea8"). InnerVolumeSpecName "kube-api-access-dcbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:19.558977 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.558955    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d" pod "a49161a2-5532-4906-92d5-33d4fd8cbc86" (UID: "a49161a2-5532-4906-92d5-33d4fd8cbc86"). InnerVolumeSpecName "kube-api-access-sx22d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:19.578642 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.578622    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:19.584804 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.584783    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:19.644550 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.644181    5419 patch_prober.go:29] interesting pod/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused" start-of-body=
Oct 31 02:00:19.644550 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.644242    5419 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.31:15021/healthz/ready\": dial tcp 10.133.1.31:15021: connect: connection refused"
Oct 31 02:00:19.661471 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.657907    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx22d\" (UniqueName: \"kubernetes.io/projected/a49161a2-5532-4906-92d5-33d4fd8cbc86-kube-api-access-sx22d\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.661471 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.657947    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbdm\" (UniqueName: \"kubernetes.io/projected/73be4d39-b90a-4ce9-b13d-03f5320e1ea8-kube-api-access-dcbdm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:19.965422 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.965386    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"]
Oct 31 02:00:19.973665 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:19.973634    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4269ad0_e14d_4773_8a5c_55f9ed43f3c0.slice/crio-04b677b84f4ec58ae05925aa734437201f5b65830fc31151ab9a9591ba6ed3f7 WatchSource:0}: Error finding container 04b677b84f4ec58ae05925aa734437201f5b65830fc31151ab9a9591ba6ed3f7: Status 404 returned error can't find the container with id 04b677b84f4ec58ae05925aa734437201f5b65830fc31151ab9a9591ba6ed3f7
Oct 31 02:00:19.998663 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:19.998631    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"]
Oct 31 02:00:20.003925 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.003899    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057e2d-29890200-9jvwp"]
Oct 31 02:00:20.012637 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.012103    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"]
Oct 31 02:00:20.015358 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:00:20.015316    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a6bfbb_99f4_459a_9518_98580ef304d0.slice/crio-0f894567326283759ee883dbeefa2f383253f3a5e86f8ca4ac6b50bfc13245c2 WatchSource:0}: Error finding container 0f894567326283759ee883dbeefa2f383253f3a5e86f8ca4ac6b50bfc13245c2: Status 404 returned error can't find the container with id 0f894567326283759ee883dbeefa2f383253f3a5e86f8ca4ac6b50bfc13245c2
Oct 31 02:00:20.218270 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.218173    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"]
Oct 31 02:00:20.227429 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.227399    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-051514-29890200-knpfc"]
Oct 31 02:00:20.247703 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.247410    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"
Oct 31 02:00:20.250162 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.247459    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8" event={"ID":"73be4d39-b90a-4ce9-b13d-03f5320e1ea8","Type":"ContainerDied","Data":"f6f9d1063dd990c705e97e2c8817ea6f273c1ba681cbd0db6093d42557d10e7c"}
Oct 31 02:00:20.250162 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.248102    5419 scope.go:122] "RemoveContainer" containerID="6fdc432b982a7a81579ad1c956ffa2999ee28a2a60ebae0f85b283830698911e"
Oct 31 02:00:20.251423 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.251394    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm" event={"ID":"a49161a2-5532-4906-92d5-33d4fd8cbc86","Type":"ContainerDied","Data":"975293acf943a2c022d9520021f0f8493b292bca9e7b084e6eaeee26f94de5dc"}
Oct 31 02:00:20.251544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.251506    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"
Oct 31 02:00:20.260416 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.259927    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p" event={"ID":"23a6bfbb-99f4-459a-9518-98580ef304d0","Type":"ContainerStarted","Data":"e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176"}
Oct 31 02:00:20.260416 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.259989    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p" event={"ID":"23a6bfbb-99f4-459a-9518-98580ef304d0","Type":"ContainerStarted","Data":"0f894567326283759ee883dbeefa2f383253f3a5e86f8ca4ac6b50bfc13245c2"}
Oct 31 02:00:20.260702 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.260685    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:20.264533 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.263276    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerDied","Data":"35a9cff36b2c8bd25b921f359ca3b3658912c32df0433b3e38edfa130666a564"}
Oct 31 02:00:20.264533 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.263280    5419 generic.go:356] "Generic (PLEG): container finished" podID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerID="35a9cff36b2c8bd25b921f359ca3b3658912c32df0433b3e38edfa130666a564" exitCode=60
Oct 31 02:00:20.264533 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.263789    5419 scope.go:122] "RemoveContainer" containerID="35a9cff36b2c8bd25b921f359ca3b3658912c32df0433b3e38edfa130666a564"
Oct 31 02:00:20.264533 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:20.264132    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx_odh-ai-gateway-infra(56b0de57-5fad-4a3f-9e06-d7899b82251f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f"
Oct 31 02:00:20.266348 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.266321    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g" event={"ID":"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0","Type":"ContainerStarted","Data":"e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6"}
Oct 31 02:00:20.266542 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.266364    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g" event={"ID":"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0","Type":"ContainerStarted","Data":"04b677b84f4ec58ae05925aa734437201f5b65830fc31151ab9a9591ba6ed3f7"}
Oct 31 02:00:20.266542 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.266415    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:20.269041 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.269018    5419 generic.go:356] "Generic (PLEG): container finished" podID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerID="fed69b724f66b632d3cd8998374e37030d6f7d43e64c5b56432292c3c95098d6" exitCode=60
Oct 31 02:00:20.269231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.269097    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerDied","Data":"fed69b724f66b632d3cd8998374e37030d6f7d43e64c5b56432292c3c95098d6"}
Oct 31 02:00:20.269621 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.269591    5419 scope.go:122] "RemoveContainer" containerID="fed69b724f66b632d3cd8998374e37030d6f7d43e64c5b56432292c3c95098d6"
Oct 31 02:00:20.269915 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:20.269890    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm_odh-ai-gateway-infra(14b838a8-fe5b-4e04-a75f-9a8db2693104)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104"
Oct 31 02:00:20.273796 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.273772    5419 scope.go:122] "RemoveContainer" containerID="88251abb4eb03fc8e8844c265ad448b3e3eb32ffa954fca54604e7fd1be7c78e"
Oct 31 02:00:20.274720 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.274688    5419 generic.go:356] "Generic (PLEG): container finished" podID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerID="f3c453ac7fdd0521469c679663c25b19988ba43edab074091bd19082e272ce7c" exitCode=60
Oct 31 02:00:20.274815 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.274738    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerDied","Data":"f3c453ac7fdd0521469c679663c25b19988ba43edab074091bd19082e272ce7c"}
Oct 31 02:00:20.275131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.275104    5419 scope.go:122] "RemoveContainer" containerID="f3c453ac7fdd0521469c679663c25b19988ba43edab074091bd19082e272ce7c"
Oct 31 02:00:20.275455 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:20.275431    5419 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5_odh-ai-gateway-infra(4ad6fe6e-82e3-483c-b421-36c3802957bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc"
Oct 31 02:00:20.285227 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.285204    5419 scope.go:122] "RemoveContainer" containerID="8742e9c18a926567bfaff1a4bb48c92470ed9627eedba12a19df11f89e92e2c4"
Oct 31 02:00:20.300656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.300627    5419 scope.go:122] "RemoveContainer" containerID="f350b6cea87ab6793740933ca6bca95c2c8e969c01bc527a4e8544f3907d253d"
Oct 31 02:00:20.319446 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.319424    5419 scope.go:122] "RemoveContainer" containerID="c192f0eed0cf5cac7132f731f26b3eac0b61345384a933b3f71c8f33a58fa531"
Oct 31 02:00:20.335688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.335665    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"]
Oct 31 02:00:20.338565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.338543    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-baacee-29890200-2qgrc"]
Oct 31 02:00:20.649308 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.649279    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:20.649695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.649667    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:20.650694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.650677    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:00:20.774798 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.774757    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"]
Oct 31 02:00:20.838812 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.838786    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f5da2d-3a9a-4e1e-bd12-e0847f2b638d" path="/var/lib/kubelet/pods/13f5da2d-3a9a-4e1e-bd12-e0847f2b638d/volumes"
Oct 31 02:00:20.839252 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.839238    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b49020-b109-4499-98e2-d39675bfe1dc" path="/var/lib/kubelet/pods/d0b49020-b109-4499-98e2-d39675bfe1dc/volumes"
Oct 31 02:00:20.839622 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.839610    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69de7d1-e1d7-4d5d-89c9-606556d6f7ec" path="/var/lib/kubelet/pods/f69de7d1-e1d7-4d5d-89c9-606556d6f7ec/volumes"
Oct 31 02:00:20.887103 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:20.887076    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"]
Oct 31 02:00:21.064469 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.064428    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"]
Oct 31 02:00:21.068092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.068062    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a8e157-29890200-lwwpm"]
Oct 31 02:00:21.117438 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.117407    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"]
Oct 31 02:00:21.178797 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.178743    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g" podStartSLOduration=3.178728234 podStartE2EDuration="3.178728234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:21.176723477 +0000 UTC m=+3342.477680065" watchObservedRunningTime="2026-10-31 02:00:21.178728234 +0000 UTC m=+3342.479684811"
Oct 31 02:00:21.356859 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.356791    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p" podStartSLOduration=2.356771333 podStartE2EDuration="2.356771333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:21.315811998 +0000 UTC m=+3342.616768575" watchObservedRunningTime="2026-10-31 02:00:21.356771333 +0000 UTC m=+3342.657727911"
Oct 31 02:00:21.469184 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.468867    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"]
Oct 31 02:00:21.482267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.482221    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5632ea-29890200-4tsm8"]
Oct 31 02:00:21.549495 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.549466    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:21.648420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.648390    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:21.650468 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.650448    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:21.689699 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.689665    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt\" (UniqueName: \"kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt\") pod \"4ad6fe6e-82e3-483c-b421-36c3802957bc\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") "
Oct 31 02:00:21.689886 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.689866    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca\") pod \"4ad6fe6e-82e3-483c-b421-36c3802957bc\" (UID: \"4ad6fe6e-82e3-483c-b421-36c3802957bc\") "
Oct 31 02:00:21.690235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.690209    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca" pod "4ad6fe6e-82e3-483c-b421-36c3802957bc" (UID: "4ad6fe6e-82e3-483c-b421-36c3802957bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.690489 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.690472    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad6fe6e-82e3-483c-b421-36c3802957bc-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.691988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.691959    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt" pod "4ad6fe6e-82e3-483c-b421-36c3802957bc" (UID: "4ad6fe6e-82e3-483c-b421-36c3802957bc"). InnerVolumeSpecName "kube-api-access-cwfvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.791640 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.791589    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk\" (UniqueName: \"kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk\") pod \"14b838a8-fe5b-4e04-a75f-9a8db2693104\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") "
Oct 31 02:00:21.791836 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.791763    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca\") pod \"14b838a8-fe5b-4e04-a75f-9a8db2693104\" (UID: \"14b838a8-fe5b-4e04-a75f-9a8db2693104\") "
Oct 31 02:00:21.791901 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.791867    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca\") pod \"56b0de57-5fad-4a3f-9e06-d7899b82251f\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") "
Oct 31 02:00:21.791977 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.791922    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg\" (UniqueName: \"kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg\") pod \"56b0de57-5fad-4a3f-9e06-d7899b82251f\" (UID: \"56b0de57-5fad-4a3f-9e06-d7899b82251f\") "
Oct 31 02:00:21.792290 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.792191    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca" pod "14b838a8-fe5b-4e04-a75f-9a8db2693104" (UID: "14b838a8-fe5b-4e04-a75f-9a8db2693104"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.792290 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.792252    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca" pod "56b0de57-5fad-4a3f-9e06-d7899b82251f" (UID: "56b0de57-5fad-4a3f-9e06-d7899b82251f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:21.792882 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.792858    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b838a8-fe5b-4e04-a75f-9a8db2693104-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.792971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.792890    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56b0de57-5fad-4a3f-9e06-d7899b82251f-maas-api-service-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.792971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.792906    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwfvt\" (UniqueName: \"kubernetes.io/projected/4ad6fe6e-82e3-483c-b421-36c3802957bc-kube-api-access-cwfvt\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.794072 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.794042    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk" pod "14b838a8-fe5b-4e04-a75f-9a8db2693104" (UID: "14b838a8-fe5b-4e04-a75f-9a8db2693104"). InnerVolumeSpecName "kube-api-access-59pfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.794198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.794173    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg" pod "56b0de57-5fad-4a3f-9e06-d7899b82251f" (UID: "56b0de57-5fad-4a3f-9e06-d7899b82251f"). InnerVolumeSpecName "kube-api-access-s5jgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:21.893847 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.893772    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jgg\" (UniqueName: \"kubernetes.io/projected/56b0de57-5fad-4a3f-9e06-d7899b82251f-kube-api-access-s5jgg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:21.893847 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:21.893798    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59pfk\" (UniqueName: \"kubernetes.io/projected/14b838a8-fe5b-4e04-a75f-9a8db2693104-kube-api-access-59pfk\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:22.362090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.362063    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"
Oct 31 02:00:22.362090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.362085    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5" event={"ID":"4ad6fe6e-82e3-483c-b421-36c3802957bc","Type":"ContainerDied","Data":"2d3870df90641f15d9ea86acaf5349f115a5d801f844192d4323a9001666e5fa"}
Oct 31 02:00:22.362734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.362179    5419 scope.go:122] "RemoveContainer" containerID="f3c453ac7fdd0521469c679663c25b19988ba43edab074091bd19082e272ce7c"
Oct 31 02:00:22.363763 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.363745    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"
Oct 31 02:00:22.363885 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.363784    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx" event={"ID":"56b0de57-5fad-4a3f-9e06-d7899b82251f","Type":"ContainerDied","Data":"bf1d83a3bf6ac98543a33dcf07425de51ac390099a214bba2badf2c27a1c8702"}
Oct 31 02:00:22.365409 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.365381    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm" event={"ID":"14b838a8-fe5b-4e04-a75f-9a8db2693104","Type":"ContainerDied","Data":"36e226a5ba8187a43b1be2935b01512ec3c4fea5ec7706896f7ff5af443b3099"}
Oct 31 02:00:22.365540 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.365452    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"
Oct 31 02:00:22.374067 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.374046    5419 scope.go:122] "RemoveContainer" containerID="35a9cff36b2c8bd25b921f359ca3b3658912c32df0433b3e38edfa130666a564"
Oct 31 02:00:22.383878 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.383862    5419 scope.go:122] "RemoveContainer" containerID="fed69b724f66b632d3cd8998374e37030d6f7d43e64c5b56432292c3c95098d6"
Oct 31 02:00:22.525855 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.525825    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"]
Oct 31 02:00:22.529926 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.529898    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-00e840-29890200-scnz5"]
Oct 31 02:00:22.637387 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.637301    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"]
Oct 31 02:00:22.641212 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.641185    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-5e916d-29890200-6z7bm"]
Oct 31 02:00:22.673461 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.673431    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"]
Oct 31 02:00:22.673652 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.673631    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy" containerID="cri-o://29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b" gracePeriod=30
Oct 31 02:00:22.748753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.748720    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"]
Oct 31 02:00:22.753591 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.753561    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c4d438-29890200-27wrx"]
Oct 31 02:00:22.838248 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.838218    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" path="/var/lib/kubelet/pods/14b838a8-fe5b-4e04-a75f-9a8db2693104/volumes"
Oct 31 02:00:22.838665 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.838651    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" path="/var/lib/kubelet/pods/4ad6fe6e-82e3-483c-b421-36c3802957bc/volumes"
Oct 31 02:00:22.839017 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.839005    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" path="/var/lib/kubelet/pods/56b0de57-5fad-4a3f-9e06-d7899b82251f/volumes"
Oct 31 02:00:22.839701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.839685    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" path="/var/lib/kubelet/pods/73be4d39-b90a-4ce9-b13d-03f5320e1ea8/volumes"
Oct 31 02:00:22.840047 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:22.840035    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" path="/var/lib/kubelet/pods/a49161a2-5532-4906-92d5-33d4fd8cbc86/volumes"
Oct 31 02:00:27.987479 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:27.987451    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 02:00:28.071500 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071461    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071529    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071568    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071616    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071652    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071890 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071730    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071890 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071817    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.071890 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071857    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.072052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071855    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:28.072052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071902    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:28.072052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.071926    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy\") pod \"069362af-682c-4fd6-af88-10532a5e459d\" (UID: \"069362af-682c-4fd6-af88-10532a5e459d\") "
Oct 31 02:00:28.072247 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072165    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:28.072247 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072181    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:28.072476 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072449    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:00:28.072772 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072749    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.072842 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072781    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.072842 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072797    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069362af-682c-4fd6-af88-10532a5e459d-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.072842 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072810    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.072842 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.072825    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.074025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.073999    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "kube-api-access-vbrj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:28.074025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.073998    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:00:28.074270 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.074055    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:00:28.074473 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.074457    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy" pod "069362af-682c-4fd6-af88-10532a5e459d" (UID: "069362af-682c-4fd6-af88-10532a5e459d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:00:28.174402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.174366    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069362af-682c-4fd6-af88-10532a5e459d-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.174402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.174405    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.174609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.174421    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069362af-682c-4fd6-af88-10532a5e459d-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.174609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.174436    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrj5\" (UniqueName: \"kubernetes.io/projected/069362af-682c-4fd6-af88-10532a5e459d-kube-api-access-vbrj5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:00:28.414338 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.414298    5419 generic.go:356] "Generic (PLEG): container finished" podID="069362af-682c-4fd6-af88-10532a5e459d" containerID="29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b" exitCode=0
Oct 31 02:00:28.414519 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.414383    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"
Oct 31 02:00:28.414519 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.414392    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" event={"ID":"069362af-682c-4fd6-af88-10532a5e459d","Type":"ContainerDied","Data":"29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b"}
Oct 31 02:00:28.414519 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.414453    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj" event={"ID":"069362af-682c-4fd6-af88-10532a5e459d","Type":"ContainerDied","Data":"e79098cc59b7fbad15f300b6545f87c402a4fd57a3999c4c6764f2f49ae35616"}
Oct 31 02:00:28.414519 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.414483    5419 scope.go:122] "RemoveContainer" containerID="29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b"
Oct 31 02:00:28.425637 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.425611    5419 scope.go:122] "RemoveContainer" containerID="29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b"
Oct 31 02:00:28.425931 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:28.425912    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b\": container with ID starting with 29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b not found: ID does not exist" containerID="29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b"
Oct 31 02:00:28.425987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.425939    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b"} err="failed to get container status \"29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b\": rpc error: code = NotFound desc = could not find container \"29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b\": container with ID starting with 29b674d7d7a9e425f7246b2d9c8aa92906e5dad1b37f6ae0ab7d71367501e40b not found: ID does not exist"
Oct 31 02:00:28.578562 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.578524    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"]
Oct 31 02:00:28.628088 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.628054    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-vnjpj"]
Oct 31 02:00:28.846653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:28.846614    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069362af-682c-4fd6-af88-10532a5e459d" path="/var/lib/kubelet/pods/069362af-682c-4fd6-af88-10532a5e459d/volumes"
Oct 31 02:00:31.357133 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:31.357102    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:00:32.368914 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:32.368879    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:00:41.441287 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.441247    5419 scope.go:122] "RemoveContainer" containerID="757b3605e20342090dfc70d40f1ce9fd6fd3064866c31c9c221bf15159cefc85"
Oct 31 02:00:41.453848 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.453827    5419 scope.go:122] "RemoveContainer" containerID="e4cf0df6e22183f89b2b8c747f57175ffb8a683e9544e04b9854bd45edf67696"
Oct 31 02:00:41.463239 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.463216    5419 scope.go:122] "RemoveContainer" containerID="81c5f98fc7f63cd64d632b0cbca9f32b6a9e5e0bd92d06e527f21a0499817445"
Oct 31 02:00:41.472189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.472171    5419 scope.go:122] "RemoveContainer" containerID="7a4533a4cbe7c7438642c1f49d10369325fa4146684dd63663422696914ef62a"
Oct 31 02:00:41.481046 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.481029    5419 scope.go:122] "RemoveContainer" containerID="b78d1594875726bc5a587516efc46649cd62a78c506bffae26d0ffdc6cbcc44d"
Oct 31 02:00:41.490694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.490672    5419 scope.go:122] "RemoveContainer" containerID="5ee1be29562a443afe0ad57f827fe49ac2588a55b45ee6c83fbc0bcdda725030"
Oct 31 02:00:41.501681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.501660    5419 scope.go:122] "RemoveContainer" containerID="312e79fe120e8f31d18b0f99d04f8aef75598b9670c2e43d1b9f2044112a3238"
Oct 31 02:00:41.511308 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.511287    5419 scope.go:122] "RemoveContainer" containerID="64ff47b52e6bc5fd71ef305c923096e56c48bb60571c523df2f47910f7d48374"
Oct 31 02:00:41.521174 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:00:41.521147    5419 scope.go:122] "RemoveContainer" containerID="8e25a584707ae2523efabbd0e02226761525fd8625f748350179a5c737d5522e"
Oct 31 02:00:44.582773 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:00:44.582728    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:01:05.769995 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.769959    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"]
Oct 31 02:01:05.770712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770693    5419 cpu_manager.go:414] "containerMap: removing container" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770714    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770724    5419 cpu_manager.go:414] "containerMap: removing container" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770730    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770738    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770742    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770751    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.770761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770755    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770767    5419 cpu_manager.go:414] "containerMap: removing container" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770772    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770788    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770793    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770804    5419 cpu_manager.go:414] "containerMap: removing container" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770808    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770815    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770819    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770836    5419 cpu_manager.go:414] "containerMap: removing container" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770840    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770846    5419 cpu_manager.go:414] "containerMap: removing container" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770850    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770861    5419 cpu_manager.go:414] "containerMap: removing container" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770865    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770877    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770881    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770888    5419 cpu_manager.go:414] "containerMap: removing container" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770891    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770972    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770979    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770986    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.770990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.770993    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771002    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771006    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="069362af-682c-4fd6-af88-10532a5e459d" containerName="istio-proxy"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771011    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771017    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771022    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771028    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771033    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771038    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771043    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771048    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49161a2-5532-4906-92d5-33d4fd8cbc86" containerName="cleanup"
Oct 31 02:01:05.771695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.771053    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:01:05.777872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.777853    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.832337 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.832299    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"]
Oct 31 02:01:05.842580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.839220    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.842580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.839291    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.842580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.839439    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.842580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.839501    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrkrt\" (UniqueName: \"kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.842580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.839527    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.842580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.839663    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.876899 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.876866    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9e3b95\"/\"openshift-service-ca.crt\""
Oct 31 02:01:05.876899 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.876872    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9e3b95\"/\"kube-root-ca.crt\""
Oct 31 02:01:05.877172 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.876932    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9e3b95\"/\"default-dockercfg-xv5dr\""
Oct 31 02:01:05.877172 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.877038    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9e3b95\"/\"auth-test-model-9e3b95-kserve-self-signed-certs\""
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.943963    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.944040    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.944164    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.944303    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrkrt\" (UniqueName: \"kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.944397    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.944563    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.946119    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.947408    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.952231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.947927    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.960190 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.960061    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:05.962028 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:05.961980    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:06.003603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:06.003571    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrkrt\" (UniqueName: \"kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt\") pod \"auth-test-model-9e3b95-kserve-755d4688fc-w4fl9\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") " pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:06.090404 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:06.090311    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:06.467323 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:06.467294    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"]
Oct 31 02:01:06.468977 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:01:06.468951    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd92daa41_f0c0_47d1_ae86_7714db897538.slice/crio-d96119dbdb996879c7d04c8c2f307ec933e556b3b7b39473ff934b344931d81b WatchSource:0}: Error finding container d96119dbdb996879c7d04c8c2f307ec933e556b3b7b39473ff934b344931d81b: Status 404 returned error can't find the container with id d96119dbdb996879c7d04c8c2f307ec933e556b3b7b39473ff934b344931d81b
Oct 31 02:01:06.703892 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:06.703851    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" event={"ID":"d92daa41-f0c0-47d1-ae86-7714db897538","Type":"ContainerStarted","Data":"534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9"}
Oct 31 02:01:06.704073 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:06.703902    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" event={"ID":"d92daa41-f0c0-47d1-ae86-7714db897538","Type":"ContainerStarted","Data":"d96119dbdb996879c7d04c8c2f307ec933e556b3b7b39473ff934b344931d81b"}
Oct 31 02:01:06.817292 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:06.817163    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" podStartSLOduration=1.817118483 podStartE2EDuration="1.817118483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:06.816743956 +0000 UTC m=+3388.117700566" watchObservedRunningTime="2026-10-31 02:01:06.817118483 +0000 UTC m=+3388.118075060"
Oct 31 02:01:07.091348 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:07.091280    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:09.347395 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:09.347363    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:01:14.582810 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:01:14.582768    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:01:16.091523 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:16.091486    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:16.105569 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:16.105546    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:16.780447 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:16.780420    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:01:41.649803 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:41.649763    5419 scope.go:122] "RemoveContainer" containerID="800b6805946999c7d4ef4f161044ca8c366aa35a5482f62977aa11bb71f87ec7"
Oct 31 02:01:44.582754 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:01:44.582716    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:01:48.330643 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:01:48.330386    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:02:14.582373 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:02:14.582339    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:02:20.688126 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.688092    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"]
Oct 31 02:02:20.689015 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.688992    5419 cpu_manager.go:414] "containerMap: removing container" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:02:20.689154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.689016    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b838a8-fe5b-4e04-a75f-9a8db2693104" containerName="cleanup"
Oct 31 02:02:20.689154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.689033    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:02:20.689154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.689048    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad6fe6e-82e3-483c-b421-36c3802957bc" containerName="cleanup"
Oct 31 02:02:20.689154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.689085    5419 cpu_manager.go:414] "containerMap: removing container" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:02:20.689154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.689093    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73be4d39-b90a-4ce9-b13d-03f5320e1ea8" containerName="cleanup"
Oct 31 02:02:20.689430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.689284    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b0de57-5fad-4a3f-9e06-d7899b82251f" containerName="cleanup"
Oct 31 02:02:20.692954 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.692935    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.748712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.748677    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"]
Oct 31 02:02:20.755254 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.755225    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.755418 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.755283    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.755504 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.755479    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.755638 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.755527    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.755638 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.755626    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.755766 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.755719    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84l5c\" (UniqueName: \"kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.794279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.794228    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c4d438\"/\"kube-root-ca.crt\""
Oct 31 02:02:20.794457 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.794294    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c4d438\"/\"openshift-service-ca.crt\""
Oct 31 02:02:20.794457 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.794315    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c4d438\"/\"default-dockercfg-pl2wq\""
Oct 31 02:02:20.794457 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.794394    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c4d438\"/\"auth-test-model-c4d438-kserve-self-signed-certs\""
Oct 31 02:02:20.872838 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.857796    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.873541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.873490    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.874110 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.874071    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.874422 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.874381    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.875854 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.875130    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.875854 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.875290    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84l5c\" (UniqueName: \"kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.876061 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.876013    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.876130 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.876117    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.876261 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.876235    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.878440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.878396    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.879664 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.879641    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:20.932708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:20.932682    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84l5c\" (UniqueName: \"kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c\") pod \"auth-test-model-c4d438-kserve-86f5f9577c-m5nbj\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") " pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:21.005760 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:21.005725    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:21.386614 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:21.386589    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"]
Oct 31 02:02:21.387681 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:02:21.387653    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07283a5c_2baf_4c5a_90dc_754fe229218d.slice/crio-7fc65d484e8276b65f1d2a01d47fd614d3329d5c3b419b6ddfeefdfb507a5b90 WatchSource:0}: Error finding container 7fc65d484e8276b65f1d2a01d47fd614d3329d5c3b419b6ddfeefdfb507a5b90: Status 404 returned error can't find the container with id 7fc65d484e8276b65f1d2a01d47fd614d3329d5c3b419b6ddfeefdfb507a5b90
Oct 31 02:02:22.249452 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:22.249417    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj" event={"ID":"07283a5c-2baf-4c5a-90dc-754fe229218d","Type":"ContainerStarted","Data":"116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6"}
Oct 31 02:02:22.249862 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:22.249475    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj" event={"ID":"07283a5c-2baf-4c5a-90dc-754fe229218d","Type":"ContainerStarted","Data":"7fc65d484e8276b65f1d2a01d47fd614d3329d5c3b419b6ddfeefdfb507a5b90"}
Oct 31 02:02:22.364941 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:22.364895    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj" podStartSLOduration=2.364879938 podStartE2EDuration="2.364879938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:22.362340188 +0000 UTC m=+3463.663296766" watchObservedRunningTime="2026-10-31 02:02:22.364879938 +0000 UTC m=+3463.665836515"
Oct 31 02:02:23.006468 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:23.006439    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:31.006632 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:31.006602    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:31.019115 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:31.019085    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:31.321753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:31.321678    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:02:32.468169 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:32.468117    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:02:36.346059 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.346022    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"]
Oct 31 02:02:36.351210 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.351190    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.395716 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.395688    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.395874 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.395757    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.395874 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.395837    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28nw\" (UniqueName: \"kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.406796 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.406759    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"]
Oct 31 02:02:36.452778 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.452750    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:02:36.452955 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.452787    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:02:36.452955 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.452846    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:02:36.454670 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.454645    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:02:36.497085 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.497023    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.497325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.497256    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.497325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.497296    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28nw\" (UniqueName: \"kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.537161 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.524015    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.557634 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.557598    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.590912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.590839    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28nw\" (UniqueName: \"kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw\") pod \"maas-api-revoke-sub-28b658cd-bwkp8\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:36.665941 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:36.665860    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:37.043179 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:37.043125    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"]
Oct 31 02:02:37.044220 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:02:37.044188    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d750ebd_3909_431d_8031_a42a152bcf2b.slice/crio-cc11bf332a96ad92da2e0b42fcb127776a3ed367c73d1ed8a01e1c4dbb2cef1e WatchSource:0}: Error finding container cc11bf332a96ad92da2e0b42fcb127776a3ed367c73d1ed8a01e1c4dbb2cef1e: Status 404 returned error can't find the container with id cc11bf332a96ad92da2e0b42fcb127776a3ed367c73d1ed8a01e1c4dbb2cef1e
Oct 31 02:02:37.356919 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:37.356887    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8" event={"ID":"6d750ebd-3909-431d-8031-a42a152bcf2b","Type":"ContainerStarted","Data":"cc11bf332a96ad92da2e0b42fcb127776a3ed367c73d1ed8a01e1c4dbb2cef1e"}
Oct 31 02:02:38.365352 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:38.365263    5419 generic.go:356] "Generic (PLEG): container finished" podID="6d750ebd-3909-431d-8031-a42a152bcf2b" containerID="798b32334e198f0eeffaa7f82c24e831921ab5942020845ac9b80b6dada154b1" exitCode=0
Oct 31 02:02:38.365352 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:38.365341    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8" event={"ID":"6d750ebd-3909-431d-8031-a42a152bcf2b","Type":"ContainerDied","Data":"798b32334e198f0eeffaa7f82c24e831921ab5942020845ac9b80b6dada154b1"}
Oct 31 02:02:39.530882 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.530856    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:39.637092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.637012    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca\") pod \"6d750ebd-3909-431d-8031-a42a152bcf2b\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") "
Oct 31 02:02:39.637283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.637124    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw\" (UniqueName: \"kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw\") pod \"6d750ebd-3909-431d-8031-a42a152bcf2b\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") "
Oct 31 02:02:39.637283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.637270    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca\") pod \"6d750ebd-3909-431d-8031-a42a152bcf2b\" (UID: \"6d750ebd-3909-431d-8031-a42a152bcf2b\") "
Oct 31 02:02:39.637452 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.637428    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca" pod "6d750ebd-3909-431d-8031-a42a152bcf2b" (UID: "6d750ebd-3909-431d-8031-a42a152bcf2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:02:39.637744 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.637727    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:39.639337 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.639314    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw" pod "6d750ebd-3909-431d-8031-a42a152bcf2b" (UID: "6d750ebd-3909-431d-8031-a42a152bcf2b"). InnerVolumeSpecName "kube-api-access-k28nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:39.639431 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.639349    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca" pod "6d750ebd-3909-431d-8031-a42a152bcf2b" (UID: "6d750ebd-3909-431d-8031-a42a152bcf2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:02:39.738952 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.738915    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28nw\" (UniqueName: \"kubernetes.io/projected/6d750ebd-3909-431d-8031-a42a152bcf2b-kube-api-access-k28nw\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:39.738952 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:39.738948    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d750ebd-3909-431d-8031-a42a152bcf2b-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:40.380480 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:40.380455    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"
Oct 31 02:02:40.380640 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:40.380498    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8" event={"ID":"6d750ebd-3909-431d-8031-a42a152bcf2b","Type":"ContainerDied","Data":"cc11bf332a96ad92da2e0b42fcb127776a3ed367c73d1ed8a01e1c4dbb2cef1e"}
Oct 31 02:02:40.380640 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:40.380540    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc11bf332a96ad92da2e0b42fcb127776a3ed367c73d1ed8a01e1c4dbb2cef1e"
Oct 31 02:02:41.679994 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:41.679961    5419 scope.go:122] "RemoveContainer" containerID="7b0a22581ac9012d574a50803dc998297960bb5b0e21e33a1e3e124cda485a04"
Oct 31 02:02:44.585151 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:02:44.585093    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:02:47.092165 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.092081    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"]
Oct 31 02:02:47.092772 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.092755    5419 cpu_manager.go:414] "containerMap: removing container" podUID="6d750ebd-3909-431d-8031-a42a152bcf2b" containerName="revoke-keys"
Oct 31 02:02:47.092846 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.092772    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d750ebd-3909-431d-8031-a42a152bcf2b" containerName="revoke-keys"
Oct 31 02:02:47.092911 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.092884    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d750ebd-3909-431d-8031-a42a152bcf2b" containerName="revoke-keys"
Oct 31 02:02:47.097784 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.097764    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.150356 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.150326    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:02:47.211200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.204931    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:02:47.211200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.205043    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:02:47.211200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.205155    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:02:47.235254 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.235220    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"]
Oct 31 02:02:47.238846 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.238807    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzlf\" (UniqueName: \"kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.239019 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.238863    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.239094 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.239014    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.340403 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.340365    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzlf\" (UniqueName: \"kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.340627 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.340417    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.340627 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.340478    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.341260 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.341238    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.343419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.343350    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.398258 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.398219    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzlf\" (UniqueName: \"kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf\") pod \"maas-api-revoke-sub-5f86e453-57zvs\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.413035 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.413008    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:47.804275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:47.804239    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"]
Oct 31 02:02:47.805340 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:02:47.805314    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829d52df_509a_4fa5_b396_b5c00ffb249b.slice/crio-7628fa040b567924902852cbd286f59b420b0a4cc61936879f52c0c0c5854443 WatchSource:0}: Error finding container 7628fa040b567924902852cbd286f59b420b0a4cc61936879f52c0c0c5854443: Status 404 returned error can't find the container with id 7628fa040b567924902852cbd286f59b420b0a4cc61936879f52c0c0c5854443
Oct 31 02:02:48.445558 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:48.445470    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs" event={"ID":"829d52df-509a-4fa5-b396-b5c00ffb249b","Type":"ContainerStarted","Data":"70fdf40fdb7e7b63e1e54ec03c5c8c880349171ae0715fd4869736b6f92aff5e"}
Oct 31 02:02:48.445558 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:48.445533    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs" event={"ID":"829d52df-509a-4fa5-b396-b5c00ffb249b","Type":"ContainerStarted","Data":"7628fa040b567924902852cbd286f59b420b0a4cc61936879f52c0c0c5854443"}
Oct 31 02:02:48.558059 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:48.557988    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs" podStartSLOduration=1.557960639 podStartE2EDuration="1.557960639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:48.557872749 +0000 UTC m=+3489.858829327" watchObservedRunningTime="2026-10-31 02:02:48.557960639 +0000 UTC m=+3489.858917226"
Oct 31 02:02:49.453644 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:49.453612    5419 generic.go:356] "Generic (PLEG): container finished" podID="829d52df-509a-4fa5-b396-b5c00ffb249b" containerID="70fdf40fdb7e7b63e1e54ec03c5c8c880349171ae0715fd4869736b6f92aff5e" exitCode=0
Oct 31 02:02:49.454027 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:49.453690    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs" event={"ID":"829d52df-509a-4fa5-b396-b5c00ffb249b","Type":"ContainerDied","Data":"70fdf40fdb7e7b63e1e54ec03c5c8c880349171ae0715fd4869736b6f92aff5e"}
Oct 31 02:02:50.626854 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.626826    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:50.782456 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.782417    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca\") pod \"829d52df-509a-4fa5-b396-b5c00ffb249b\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") "
Oct 31 02:02:50.782662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.782578    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca\") pod \"829d52df-509a-4fa5-b396-b5c00ffb249b\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") "
Oct 31 02:02:50.782713 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.782670    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf\" (UniqueName: \"kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf\") pod \"829d52df-509a-4fa5-b396-b5c00ffb249b\" (UID: \"829d52df-509a-4fa5-b396-b5c00ffb249b\") "
Oct 31 02:02:50.782824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.782805    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca" pod "829d52df-509a-4fa5-b396-b5c00ffb249b" (UID: "829d52df-509a-4fa5-b396-b5c00ffb249b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:02:50.783206 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.783182    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:50.784811 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.784787    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca" pod "829d52df-509a-4fa5-b396-b5c00ffb249b" (UID: "829d52df-509a-4fa5-b396-b5c00ffb249b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:02:50.784927 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.784889    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf" pod "829d52df-509a-4fa5-b396-b5c00ffb249b" (UID: "829d52df-509a-4fa5-b396-b5c00ffb249b"). InnerVolumeSpecName "kube-api-access-gmzlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:02:50.884463 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.884422    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/829d52df-509a-4fa5-b396-b5c00ffb249b-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:50.884463 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:50.884458    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzlf\" (UniqueName: \"kubernetes.io/projected/829d52df-509a-4fa5-b396-b5c00ffb249b-kube-api-access-gmzlf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:02:51.470734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:51.470702    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"
Oct 31 02:02:51.470734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:51.470722    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs" event={"ID":"829d52df-509a-4fa5-b396-b5c00ffb249b","Type":"ContainerDied","Data":"7628fa040b567924902852cbd286f59b420b0a4cc61936879f52c0c0c5854443"}
Oct 31 02:02:51.470969 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:02:51.470780    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7628fa040b567924902852cbd286f59b420b0a4cc61936879f52c0c0c5854443"
Oct 31 02:03:14.583178 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:03:14.583114    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:03:27.933401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:27.933366    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:03:33.503318 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:33.503284    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"]
Oct 31 02:03:41.708419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:41.708383    5419 scope.go:122] "RemoveContainer" containerID="7a902dc1d269bdadfabd84376e7c239f3f43b77b09d45879a426a9cf87dab0de"
Oct 31 02:03:41.718870 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:41.718849    5419 scope.go:122] "RemoveContainer" containerID="1eb0a1edd85c43223a4d965204c47d476fbdfaccf7199de43173ddb01bb319e0"
Oct 31 02:03:44.591889 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:03:44.591851    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:03:48.524648 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.524603    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd" podUID="de303017-0fcc-48aa-aaef-b3fcd150735b" containerName="main" containerID="cri-o://4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18" gracePeriod=45
Oct 31 02:03:48.800037 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.800007    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 02:03:48.855942 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.855912    5419 generic.go:356] "Generic (PLEG): container finished" podID="de303017-0fcc-48aa-aaef-b3fcd150735b" containerID="4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18" exitCode=0
Oct 31 02:03:48.856155 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.855948    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"
Oct 31 02:03:48.856155 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.855989    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd" event={"ID":"de303017-0fcc-48aa-aaef-b3fcd150735b","Type":"ContainerDied","Data":"4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18"}
Oct 31 02:03:48.856155 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.856033    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd" event={"ID":"de303017-0fcc-48aa-aaef-b3fcd150735b","Type":"ContainerDied","Data":"be6cb640375cbac4b90b87fcc037578645f159774e80b657078522415becea8f"}
Oct 31 02:03:48.856155 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.856057    5419 scope.go:122] "RemoveContainer" containerID="4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18"
Oct 31 02:03:48.865835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.865812    5419 scope.go:122] "RemoveContainer" containerID="4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18"
Oct 31 02:03:48.866127 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:03:48.866107    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18\": container with ID starting with 4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18 not found: ID does not exist" containerID="4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18"
Oct 31 02:03:48.866222 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.866149    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18"} err="failed to get container status \"4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18\": rpc error: code = NotFound desc = could not find container \"4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18\": container with ID starting with 4a37cbf2ea393eda22db0f27ba985be0412482a8280231af169784ce0cb5ba18 not found: ID does not exist"
Oct 31 02:03:48.882968 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.882946    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache\") pod \"de303017-0fcc-48aa-aaef-b3fcd150735b\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") "
Oct 31 02:03:48.883079 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883029    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm\") pod \"de303017-0fcc-48aa-aaef-b3fcd150735b\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") "
Oct 31 02:03:48.883173 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883111    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home\") pod \"de303017-0fcc-48aa-aaef-b3fcd150735b\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") "
Oct 31 02:03:48.883242 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883170    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn\" (UniqueName: \"kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn\") pod \"de303017-0fcc-48aa-aaef-b3fcd150735b\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") "
Oct 31 02:03:48.883242 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883217    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir\") pod \"de303017-0fcc-48aa-aaef-b3fcd150735b\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") "
Oct 31 02:03:48.883242 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883220    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache" pod "de303017-0fcc-48aa-aaef-b3fcd150735b" (UID: "de303017-0fcc-48aa-aaef-b3fcd150735b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:48.883415 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883265    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs\" (UniqueName: \"kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs\") pod \"de303017-0fcc-48aa-aaef-b3fcd150735b\" (UID: \"de303017-0fcc-48aa-aaef-b3fcd150735b\") "
Oct 31 02:03:48.883415 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883374    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home" pod "de303017-0fcc-48aa-aaef-b3fcd150735b" (UID: "de303017-0fcc-48aa-aaef-b3fcd150735b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:48.883609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.883585    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir" pod "de303017-0fcc-48aa-aaef-b3fcd150735b" (UID: "de303017-0fcc-48aa-aaef-b3fcd150735b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:48.884090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.884065    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:48.884233 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.884095    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:48.884233 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.884110    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:48.885406 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.885386    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm" pod "de303017-0fcc-48aa-aaef-b3fcd150735b" (UID: "de303017-0fcc-48aa-aaef-b3fcd150735b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:03:48.885933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.885900    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs" pod "de303017-0fcc-48aa-aaef-b3fcd150735b" (UID: "de303017-0fcc-48aa-aaef-b3fcd150735b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:03:48.885933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.885921    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn" pod "de303017-0fcc-48aa-aaef-b3fcd150735b" (UID: "de303017-0fcc-48aa-aaef-b3fcd150735b"). InnerVolumeSpecName "kube-api-access-6mksn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:03:48.985545 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.985516    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de303017-0fcc-48aa-aaef-b3fcd150735b-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:48.985545 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.985541    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de303017-0fcc-48aa-aaef-b3fcd150735b-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:48.985545 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:48.985551    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mksn\" (UniqueName: \"kubernetes.io/projected/de303017-0fcc-48aa-aaef-b3fcd150735b-kube-api-access-6mksn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:03:49.287300 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:49.285431    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"]
Oct 31 02:03:49.307200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:49.306449    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5e916d/auth-test-model-5e916d-kserve-57957cd989-8p7zd"]
Oct 31 02:03:50.837659 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:03:50.837630    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de303017-0fcc-48aa-aaef-b3fcd150735b" path="/var/lib/kubelet/pods/de303017-0fcc-48aa-aaef-b3fcd150735b/volumes"
Oct 31 02:04:12.082577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:12.082536    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"]
Oct 31 02:04:12.083221 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:12.082773    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy" containerID="cri-o://169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901" gracePeriod=30
Oct 31 02:04:14.582877 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:04:14.582835    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:04:17.360913 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.360853    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 02:04:17.443824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.443791    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.443824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.443837    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444104 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.443909    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444104 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.443994    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444104 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444022    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444104 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444070    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444104 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444079    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.444399 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444123    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444399 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444196    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444399 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444260    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs\") pod \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\" (UID: \"c2c9953f-0348-4ca0-a0e5-b0d4e5226580\") "
Oct 31 02:04:17.444399 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444271    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.444610 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444405    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.444610 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444429    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:17.444843 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444822    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.444907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444850    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.444907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444865    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.444907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444878    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.444907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.444888    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.446431 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.446394    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "kube-api-access-x2ffs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:17.446855 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.446832    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:17.447095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.447075    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:04:17.447248 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.447228    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo" pod "c2c9953f-0348-4ca0-a0e5-b0d4e5226580" (UID: "c2c9953f-0348-4ca0-a0e5-b0d4e5226580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:04:17.546669 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.546628    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.546852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.546659    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2ffs\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-kube-api-access-x2ffs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.546852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.546769    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.546852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.546781    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:17.546852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:17.546794    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c9953f-0348-4ca0-a0e5-b0d4e5226580-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:18.111495 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.111460    5419 generic.go:356] "Generic (PLEG): container finished" podID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerID="169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901" exitCode=0
Oct 31 02:04:18.111694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.111528    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"
Oct 31 02:04:18.111694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.111538    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" event={"ID":"c2c9953f-0348-4ca0-a0e5-b0d4e5226580","Type":"ContainerDied","Data":"169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901"}
Oct 31 02:04:18.111694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.111602    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s" event={"ID":"c2c9953f-0348-4ca0-a0e5-b0d4e5226580","Type":"ContainerDied","Data":"b68aca2ec0676b9bbeb0a1fdcc1aab8ef5962cf2fb7383b0ec74544fab7dc7e4"}
Oct 31 02:04:18.111694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.111624    5419 scope.go:122] "RemoveContainer" containerID="169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901"
Oct 31 02:04:18.123118 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.123102    5419 scope.go:122] "RemoveContainer" containerID="169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901"
Oct 31 02:04:18.123443 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:04:18.123422    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901\": container with ID starting with 169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901 not found: ID does not exist" containerID="169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901"
Oct 31 02:04:18.123504 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.123453    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901"} err="failed to get container status \"169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901\": rpc error: code = NotFound desc = could not find container \"169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901\": container with ID starting with 169593d1b0d4af99f7dac03f4f7cbf845f317d3048acef4cee403a57990d4901 not found: ID does not exist"
Oct 31 02:04:18.276095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.276063    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"]
Oct 31 02:04:18.281736 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.281711    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a8e157-openshift-default-7fb96689c7-bv25s"]
Oct 31 02:04:18.838120 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:18.838089    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" path="/var/lib/kubelet/pods/c2c9953f-0348-4ca0-a0e5-b0d4e5226580/volumes"
Oct 31 02:04:26.153534 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:26.153493    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:04:37.038008 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.037973    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"]
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038648    5419 cpu_manager.go:414] "containerMap: removing container" podUID="829d52df-509a-4fa5-b396-b5c00ffb249b" containerName="revoke-keys"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038662    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829d52df-509a-4fa5-b396-b5c00ffb249b" containerName="revoke-keys"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038672    5419 cpu_manager.go:414] "containerMap: removing container" podUID="de303017-0fcc-48aa-aaef-b3fcd150735b" containerName="main"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038677    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de303017-0fcc-48aa-aaef-b3fcd150735b" containerName="main"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038701    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038707    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038802    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c9953f-0348-4ca0-a0e5-b0d4e5226580" containerName="istio-proxy"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038818    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="829d52df-509a-4fa5-b396-b5c00ffb249b" containerName="revoke-keys"
Oct 31 02:04:37.040368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.038824    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="de303017-0fcc-48aa-aaef-b3fcd150735b" containerName="main"
Oct 31 02:04:37.042346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.042330    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.099286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.099249    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"]
Oct 31 02:04:37.102495 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.102466    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.102633 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.102586    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bq6h\" (UniqueName: \"kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.102757 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.102738    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.144584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.144554    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:04:37.144584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.144574    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:04:37.144799 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.144593    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:04:37.144799 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.144631    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:04:37.203709 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.203673    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bq6h\" (UniqueName: \"kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.203878 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.203853    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.203983 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.203966    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.204747 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.204717    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.206631 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.206606    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.260115 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.260088    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bq6h\" (UniqueName: \"kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h\") pod \"maas-api-revoke-keys-e2e-explicit-c4be41-chtq2\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.355657 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.355561    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:37.735531 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:37.735505    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"]
Oct 31 02:04:37.736950 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:04:37.736924    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea73f1b_fea1_4894_b053_a1b77dc30b46.slice/crio-311b3f869c0bbfe3e7d0e013ed8623f67371532db73be1786a10398a2be4a4cc WatchSource:0}: Error finding container 311b3f869c0bbfe3e7d0e013ed8623f67371532db73be1786a10398a2be4a4cc: Status 404 returned error can't find the container with id 311b3f869c0bbfe3e7d0e013ed8623f67371532db73be1786a10398a2be4a4cc
Oct 31 02:04:38.250690 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:38.250591    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2" event={"ID":"fea73f1b-fea1-4894-b053-a1b77dc30b46","Type":"ContainerStarted","Data":"5243f75c3fbe40415fa2fca1435bcfa140500d10b094d307232bad13781895f8"}
Oct 31 02:04:38.251071 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:38.250692    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2" event={"ID":"fea73f1b-fea1-4894-b053-a1b77dc30b46","Type":"ContainerStarted","Data":"311b3f869c0bbfe3e7d0e013ed8623f67371532db73be1786a10398a2be4a4cc"}
Oct 31 02:04:38.363316 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:38.363262    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2" podStartSLOduration=2.3632459089999998 podStartE2EDuration="2.363245909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:04:38.362606545 +0000 UTC m=+3599.663563124" watchObservedRunningTime="2026-10-31 02:04:38.363245909 +0000 UTC m=+3599.664202487"
Oct 31 02:04:39.258477 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:39.258440    5419 generic.go:356] "Generic (PLEG): container finished" podID="fea73f1b-fea1-4894-b053-a1b77dc30b46" containerID="5243f75c3fbe40415fa2fca1435bcfa140500d10b094d307232bad13781895f8" exitCode=0
Oct 31 02:04:39.258916 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:39.258487    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2" event={"ID":"fea73f1b-fea1-4894-b053-a1b77dc30b46","Type":"ContainerDied","Data":"5243f75c3fbe40415fa2fca1435bcfa140500d10b094d307232bad13781895f8"}
Oct 31 02:04:40.045403 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.045363    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"]
Oct 31 02:04:40.047990 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.047966    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fded139e-2nhr2"]
Oct 31 02:04:40.426504 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.426475    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:40.542917 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.542883    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca\") pod \"fea73f1b-fea1-4894-b053-a1b77dc30b46\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") "
Oct 31 02:04:40.543123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.543100    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca\") pod \"fea73f1b-fea1-4894-b053-a1b77dc30b46\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") "
Oct 31 02:04:40.543229 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.543213    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h\" (UniqueName: \"kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h\") pod \"fea73f1b-fea1-4894-b053-a1b77dc30b46\" (UID: \"fea73f1b-fea1-4894-b053-a1b77dc30b46\") "
Oct 31 02:04:40.543534 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.543504    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca" pod "fea73f1b-fea1-4894-b053-a1b77dc30b46" (UID: "fea73f1b-fea1-4894-b053-a1b77dc30b46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:40.543973 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.543951    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:40.545324 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.545300    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca" pod "fea73f1b-fea1-4894-b053-a1b77dc30b46" (UID: "fea73f1b-fea1-4894-b053-a1b77dc30b46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:04:40.545783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.545757    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h" pod "fea73f1b-fea1-4894-b053-a1b77dc30b46" (UID: "fea73f1b-fea1-4894-b053-a1b77dc30b46"). InnerVolumeSpecName "kube-api-access-4bq6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:40.644844 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.644756    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea73f1b-fea1-4894-b053-a1b77dc30b46-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:40.644844 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.644789    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bq6h\" (UniqueName: \"kubernetes.io/projected/fea73f1b-fea1-4894-b053-a1b77dc30b46-kube-api-access-4bq6h\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:40.837862 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:40.837829    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a3f596-0287-4780-aac0-76fe95d3b22e" path="/var/lib/kubelet/pods/07a3f596-0287-4780-aac0-76fe95d3b22e/volumes"
Oct 31 02:04:41.272170 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:41.272118    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"
Oct 31 02:04:41.272170 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:41.272151    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2" event={"ID":"fea73f1b-fea1-4894-b053-a1b77dc30b46","Type":"ContainerDied","Data":"311b3f869c0bbfe3e7d0e013ed8623f67371532db73be1786a10398a2be4a4cc"}
Oct 31 02:04:41.272436 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:41.272206    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311b3f869c0bbfe3e7d0e013ed8623f67371532db73be1786a10398a2be4a4cc"
Oct 31 02:04:44.584089 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:04:44.584055    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:04:48.556931 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:48.556813    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"]
Oct 31 02:04:48.661174 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:48.661116    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"]
Oct 31 02:04:51.046974 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:51.046942    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"]
Oct 31 02:04:51.096282 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:51.096250    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3140eac-2n8lt"]
Oct 31 02:04:52.837933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:52.837900    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e774ae5-b8ae-4ad3-a5aa-8628814003a0" path="/var/lib/kubelet/pods/4e774ae5-b8ae-4ad3-a5aa-8628814003a0/volumes"
Oct 31 02:04:53.557524 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.557471    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p" podUID="23a6bfbb-99f4-459a-9518-98580ef304d0" containerName="payload-processing" containerID="cri-o://e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176" gracePeriod=25
Oct 31 02:04:53.661873 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.661836    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g" podUID="e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" containerName="payload-pre-processing" containerID="cri-o://e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6" gracePeriod=25
Oct 31 02:04:53.822335 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.822208    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:04:53.871725 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.871690    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn\" (UniqueName: \"kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn\") pod \"23a6bfbb-99f4-459a-9518-98580ef304d0\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") "
Oct 31 02:04:53.872103 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.871838    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume\") pod \"23a6bfbb-99f4-459a-9518-98580ef304d0\" (UID: \"23a6bfbb-99f4-459a-9518-98580ef304d0\") "
Oct 31 02:04:53.872780 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.872748    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume" pod "23a6bfbb-99f4-459a-9518-98580ef304d0" (UID: "23a6bfbb-99f4-459a-9518-98580ef304d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:53.875630 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.875593    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn" pod "23a6bfbb-99f4-459a-9518-98580ef304d0" (UID: "23a6bfbb-99f4-459a-9518-98580ef304d0"). InnerVolumeSpecName "kube-api-access-fl4tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:53.929961 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.929937    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:04:53.973306 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.973268    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w\" (UniqueName: \"kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w\") pod \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") "
Oct 31 02:04:53.973499 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.973485    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume\") pod \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\" (UID: \"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0\") "
Oct 31 02:04:53.973988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.973903    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume" pod "e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" (UID: "e4269ad0-e14d-4773-8a5c-55f9ed43f3c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:04:53.973988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.973969    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl4tn\" (UniqueName: \"kubernetes.io/projected/23a6bfbb-99f4-459a-9518-98580ef304d0-kube-api-access-fl4tn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:53.973988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.973991    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a6bfbb-99f4-459a-9518-98580ef304d0-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:53.975891 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:53.975867    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w" pod "e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" (UID: "e4269ad0-e14d-4773-8a5c-55f9ed43f3c0"). InnerVolumeSpecName "kube-api-access-k969w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:04:54.074854 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.074779    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k969w\" (UniqueName: \"kubernetes.io/projected/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-kube-api-access-k969w\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:54.074854 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.074805    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:04:54.386362 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.386280    5419 generic.go:356] "Generic (PLEG): container finished" podID="23a6bfbb-99f4-459a-9518-98580ef304d0" containerID="e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176" exitCode=0
Oct 31 02:04:54.386362 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.386330    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"
Oct 31 02:04:54.386583 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.386359    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p" event={"ID":"23a6bfbb-99f4-459a-9518-98580ef304d0","Type":"ContainerDied","Data":"e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176"}
Oct 31 02:04:54.386583 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.386411    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p" event={"ID":"23a6bfbb-99f4-459a-9518-98580ef304d0","Type":"ContainerDied","Data":"0f894567326283759ee883dbeefa2f383253f3a5e86f8ca4ac6b50bfc13245c2"}
Oct 31 02:04:54.386583 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.386436    5419 scope.go:122] "RemoveContainer" containerID="e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176"
Oct 31 02:04:54.387867 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.387842    5419 generic.go:356] "Generic (PLEG): container finished" podID="e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" containerID="e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6" exitCode=0
Oct 31 02:04:54.387999 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.387885    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g" event={"ID":"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0","Type":"ContainerDied","Data":"e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6"}
Oct 31 02:04:54.387999 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.387898    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"
Oct 31 02:04:54.387999 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.387923    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g" event={"ID":"e4269ad0-e14d-4773-8a5c-55f9ed43f3c0","Type":"ContainerDied","Data":"04b677b84f4ec58ae05925aa734437201f5b65830fc31151ab9a9591ba6ed3f7"}
Oct 31 02:04:54.412988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.412958    5419 scope.go:122] "RemoveContainer" containerID="e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176"
Oct 31 02:04:54.413526 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:04:54.413502    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176\": container with ID starting with e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176 not found: ID does not exist" containerID="e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176"
Oct 31 02:04:54.413632 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.413534    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176"} err="failed to get container status \"e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176\": rpc error: code = NotFound desc = could not find container \"e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176\": container with ID starting with e87f26d81163b90c983a753dab7dd7509416c7b2dd880344be2d2a5e02fec176 not found: ID does not exist"
Oct 31 02:04:54.413632 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.413552    5419 scope.go:122] "RemoveContainer" containerID="e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6"
Oct 31 02:04:54.422942 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.422922    5419 scope.go:122] "RemoveContainer" containerID="e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6"
Oct 31 02:04:54.423263 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:04:54.423241    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6\": container with ID starting with e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6 not found: ID does not exist" containerID="e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6"
Oct 31 02:04:54.423328 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.423274    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6"} err="failed to get container status \"e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6\": rpc error: code = NotFound desc = could not find container \"e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6\": container with ID starting with e0d2e285835fd932003b9a783232bb474dd5e74e51c1327704321118935a4df6 not found: ID does not exist"
Oct 31 02:04:54.548969 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.548935    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"]
Oct 31 02:04:54.551700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.551674    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c4be41-ccb87495-zm97g"]
Oct 31 02:04:54.705485 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.705395    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"]
Oct 31 02:04:54.709086 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.709060    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c4be41-5bd8d689f9-4275p"]
Oct 31 02:04:54.838380 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.838343    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a6bfbb-99f4-459a-9518-98580ef304d0" path="/var/lib/kubelet/pods/23a6bfbb-99f4-459a-9518-98580ef304d0/volumes"
Oct 31 02:04:54.838766 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:54.838753    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" path="/var/lib/kubelet/pods/e4269ad0-e14d-4773-8a5c-55f9ed43f3c0/volumes"
Oct 31 02:04:58.641917 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:58.641828    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"]
Oct 31 02:04:58.691002 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:58.690972    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c4be41-chtq2"]
Oct 31 02:04:58.839633 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:04:58.839593    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea73f1b-fea1-4894-b053-a1b77dc30b46" path="/var/lib/kubelet/pods/fea73f1b-fea1-4894-b053-a1b77dc30b46/volumes"
Oct 31 02:05:06.034261 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:06.034228    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:05:14.582303 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:05:14.582258    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:05:16.153641 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.153605    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"]
Oct 31 02:05:16.154355 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154335    5419 cpu_manager.go:414] "containerMap: removing container" podUID="23a6bfbb-99f4-459a-9518-98580ef304d0" containerName="payload-processing"
Oct 31 02:05:16.154355 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154355    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a6bfbb-99f4-459a-9518-98580ef304d0" containerName="payload-processing"
Oct 31 02:05:16.154466 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154370    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fea73f1b-fea1-4894-b053-a1b77dc30b46" containerName="revoke-keys"
Oct 31 02:05:16.154466 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154375    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea73f1b-fea1-4894-b053-a1b77dc30b46" containerName="revoke-keys"
Oct 31 02:05:16.154466 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154394    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" containerName="payload-pre-processing"
Oct 31 02:05:16.154466 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154409    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" containerName="payload-pre-processing"
Oct 31 02:05:16.154603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154510    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a6bfbb-99f4-459a-9518-98580ef304d0" containerName="payload-processing"
Oct 31 02:05:16.154603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154522    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea73f1b-fea1-4894-b053-a1b77dc30b46" containerName="revoke-keys"
Oct 31 02:05:16.154603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.154532    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4269ad0-e14d-4773-8a5c-55f9ed43f3c0" containerName="payload-pre-processing"
Oct 31 02:05:16.158010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.157994    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.278500 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.277361    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.278905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.278865    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.279190 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.279170    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.279414 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.279395    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkftp\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.279636 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.279619    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.279866 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.279833    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.280047 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.280032    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.280212 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.280197    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.280345 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.280329    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.283286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.283257    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8114c9-openshift-default-dockercfg-j7tqn\""
Oct 31 02:05:16.293586 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.292679    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"]
Oct 31 02:05:16.383157 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.382752    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383157 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.382834    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383157 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.382875    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383157 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.382939    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383157 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.383011    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383157 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.383111    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.383175    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkftp\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.383206    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.383514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.383267    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.386031 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.383759    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.386031 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.384452    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.386031 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.384698    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.386031 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.384948    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.386747 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.386723    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.388031 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.388010    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.389269 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.389247    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.443470 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.443398    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.443470 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.443462    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkftp\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp\") pod \"e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") " pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.473299 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.472380    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:16.925541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.925516    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"]
Oct 31 02:05:16.927560 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:05:16.927527    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7da9ad5_4039_429b_96cc_14f051bda03d.slice/crio-824ea09a4c17e46163420e5d03eef9de14b943b47c35e248dfef384497f78966 WatchSource:0}: Error finding container 824ea09a4c17e46163420e5d03eef9de14b943b47c35e248dfef384497f78966: Status 404 returned error can't find the container with id 824ea09a4c17e46163420e5d03eef9de14b943b47c35e248dfef384497f78966
Oct 31 02:05:16.930165 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.930111    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 02:05:16.930283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.930199    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 02:05:16.930283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:16.930229    5419 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"}
Oct 31 02:05:17.421932 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.421898    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"]
Oct 31 02:05:17.426063 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.426046    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.478358 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.478326    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:05:17.478358 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.478356    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:05:17.478358 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.478391    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:05:17.481962 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.481940    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"]
Oct 31 02:05:17.501039 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.501008    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cbnq\" (UniqueName: \"kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.501289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.501266    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.501363 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.501321    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.527750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.527720    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:05:17.554119 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.554087    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" event={"ID":"c7da9ad5-4039-429b-96cc-14f051bda03d","Type":"ContainerStarted","Data":"9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740"}
Oct 31 02:05:17.554305 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.554163    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" event={"ID":"c7da9ad5-4039-429b-96cc-14f051bda03d","Type":"ContainerStarted","Data":"824ea09a4c17e46163420e5d03eef9de14b943b47c35e248dfef384497f78966"}
Oct 31 02:05:17.602366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.602333    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.602576 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.602386    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.602576 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.602516    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cbnq\" (UniqueName: \"kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.603225 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.603202    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.604988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.604969    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.658773 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.658743    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cbnq\" (UniqueName: \"kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq\") pod \"maas-api-revoke-sub-1ee2c2b5-tdhfn\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:17.674636 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.674542    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" podStartSLOduration=1.674528446 podStartE2EDuration="1.674528446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:17.671705955 +0000 UTC m=+3638.972662533" watchObservedRunningTime="2026-10-31 02:05:17.674528446 +0000 UTC m=+3638.975485023"
Oct 31 02:05:17.740370 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:17.740335    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:18.161740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:18.161712    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"]
Oct 31 02:05:18.162268 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:05:18.162244    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c3fd9e_0272_4714_a17b_8f3556235ef5.slice/crio-3931f1a48d37bae1ec28b54c0237af235fe1a194da7d1d8a79728886989eeb23 WatchSource:0}: Error finding container 3931f1a48d37bae1ec28b54c0237af235fe1a194da7d1d8a79728886989eeb23: Status 404 returned error can't find the container with id 3931f1a48d37bae1ec28b54c0237af235fe1a194da7d1d8a79728886989eeb23
Oct 31 02:05:18.477436 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:18.477401    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:18.483883 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:18.483856    5419 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:18.563588 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:18.563552    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn" event={"ID":"11c3fd9e-0272-4714-a17b-8f3556235ef5","Type":"ContainerStarted","Data":"3931f1a48d37bae1ec28b54c0237af235fe1a194da7d1d8a79728886989eeb23"}
Oct 31 02:05:18.563820 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:18.563795    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:18.564828 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:18.564806    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:05:19.571740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:19.571702    5419 generic.go:356] "Generic (PLEG): container finished" podID="11c3fd9e-0272-4714-a17b-8f3556235ef5" containerID="3e8c93024717b44665591399906f1a8a0dd720b77d4f7336eb97cf3af654d5b5" exitCode=0
Oct 31 02:05:19.572176 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:19.571784    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn" event={"ID":"11c3fd9e-0272-4714-a17b-8f3556235ef5","Type":"ContainerDied","Data":"3e8c93024717b44665591399906f1a8a0dd720b77d4f7336eb97cf3af654d5b5"}
Oct 31 02:05:20.740938 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.740911    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:20.844614 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.844538    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca\") pod \"11c3fd9e-0272-4714-a17b-8f3556235ef5\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") "
Oct 31 02:05:20.844614 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.844603    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca\") pod \"11c3fd9e-0272-4714-a17b-8f3556235ef5\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") "
Oct 31 02:05:20.844914 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.844779    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq\" (UniqueName: \"kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq\") pod \"11c3fd9e-0272-4714-a17b-8f3556235ef5\" (UID: \"11c3fd9e-0272-4714-a17b-8f3556235ef5\") "
Oct 31 02:05:20.844914 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.844900    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca" pod "11c3fd9e-0272-4714-a17b-8f3556235ef5" (UID: "11c3fd9e-0272-4714-a17b-8f3556235ef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:20.845673 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.845650    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.847010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.846980    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca" pod "11c3fd9e-0272-4714-a17b-8f3556235ef5" (UID: "11c3fd9e-0272-4714-a17b-8f3556235ef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:20.847133 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.847117    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq" pod "11c3fd9e-0272-4714-a17b-8f3556235ef5" (UID: "11c3fd9e-0272-4714-a17b-8f3556235ef5"). InnerVolumeSpecName "kube-api-access-6cbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:20.947334 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.947305    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c3fd9e-0272-4714-a17b-8f3556235ef5-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:20.947334 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:20.947333    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cbnq\" (UniqueName: \"kubernetes.io/projected/11c3fd9e-0272-4714-a17b-8f3556235ef5-kube-api-access-6cbnq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:21.586195 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:21.586131    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn" event={"ID":"11c3fd9e-0272-4714-a17b-8f3556235ef5","Type":"ContainerDied","Data":"3931f1a48d37bae1ec28b54c0237af235fe1a194da7d1d8a79728886989eeb23"}
Oct 31 02:05:21.586377 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:21.586216    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3931f1a48d37bae1ec28b54c0237af235fe1a194da7d1d8a79728886989eeb23"
Oct 31 02:05:21.586377 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:21.586177    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"
Oct 31 02:05:28.170571 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.170538    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"]
Oct 31 02:05:28.171153 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.171123    5419 cpu_manager.go:414] "containerMap: removing container" podUID="11c3fd9e-0272-4714-a17b-8f3556235ef5" containerName="revoke-keys"
Oct 31 02:05:28.171209 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.171154    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c3fd9e-0272-4714-a17b-8f3556235ef5" containerName="revoke-keys"
Oct 31 02:05:28.171268 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.171259    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c3fd9e-0272-4714-a17b-8f3556235ef5" containerName="revoke-keys"
Oct 31 02:05:28.174700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.174682    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.255231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.255202    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.255406 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.255373    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.255452 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.255424    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8dsl\" (UniqueName: \"kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.276643 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.276611    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:05:28.276814 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.276636    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:05:28.276814 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.276656    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:05:28.276814 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.276686    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:05:28.279757 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.279735    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"]
Oct 31 02:05:28.356452 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.356423    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.356672 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.356470    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8dsl\" (UniqueName: \"kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.356672 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.356547    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.357191 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.357131    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.359004 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.358985    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.413045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.413014    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8dsl\" (UniqueName: \"kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl\") pod \"maas-api-revoke-sub-d66ffa8d-m5b85\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.487966 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.487939    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:28.866971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:28.866949    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"]
Oct 31 02:05:28.868542 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:05:28.868514    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4314f51_8cae_40ef_a986_942812e3210e.slice/crio-c1445113d9c50325cfe153199703cfe7dd14b8721ef4ab7fdc877b27c24ebe44 WatchSource:0}: Error finding container c1445113d9c50325cfe153199703cfe7dd14b8721ef4ab7fdc877b27c24ebe44: Status 404 returned error can't find the container with id c1445113d9c50325cfe153199703cfe7dd14b8721ef4ab7fdc877b27c24ebe44
Oct 31 02:05:29.642987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:29.642953    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85" event={"ID":"f4314f51-8cae-40ef-a986-942812e3210e","Type":"ContainerStarted","Data":"9c8bdff65742ac5bda369407da257b1856480286b211d5af55b0fd88e92c0044"}
Oct 31 02:05:29.643497 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:29.643007    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85" event={"ID":"f4314f51-8cae-40ef-a986-942812e3210e","Type":"ContainerStarted","Data":"c1445113d9c50325cfe153199703cfe7dd14b8721ef4ab7fdc877b27c24ebe44"}
Oct 31 02:05:29.758614 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:29.758558    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85" podStartSLOduration=1.7585414670000001 podStartE2EDuration="1.758541467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:29.755958624 +0000 UTC m=+3651.056915200" watchObservedRunningTime="2026-10-31 02:05:29.758541467 +0000 UTC m=+3651.059498043"
Oct 31 02:05:30.650677 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:30.650641    5419 generic.go:356] "Generic (PLEG): container finished" podID="f4314f51-8cae-40ef-a986-942812e3210e" containerID="9c8bdff65742ac5bda369407da257b1856480286b211d5af55b0fd88e92c0044" exitCode=0
Oct 31 02:05:30.651070 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:30.650690    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85" event={"ID":"f4314f51-8cae-40ef-a986-942812e3210e","Type":"ContainerDied","Data":"9c8bdff65742ac5bda369407da257b1856480286b211d5af55b0fd88e92c0044"}
Oct 31 02:05:31.811000 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.810974    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:31.900557 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.900526    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca\") pod \"f4314f51-8cae-40ef-a986-942812e3210e\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") "
Oct 31 02:05:31.900738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.900571    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca\") pod \"f4314f51-8cae-40ef-a986-942812e3210e\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") "
Oct 31 02:05:31.900738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.900653    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl\" (UniqueName: \"kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl\") pod \"f4314f51-8cae-40ef-a986-942812e3210e\" (UID: \"f4314f51-8cae-40ef-a986-942812e3210e\") "
Oct 31 02:05:31.900965 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.900932    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca" pod "f4314f51-8cae-40ef-a986-942812e3210e" (UID: "f4314f51-8cae-40ef-a986-942812e3210e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:31.901560 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.901537    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:31.903099 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.903076    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca" pod "f4314f51-8cae-40ef-a986-942812e3210e" (UID: "f4314f51-8cae-40ef-a986-942812e3210e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:05:31.903233 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:31.903108    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl" pod "f4314f51-8cae-40ef-a986-942812e3210e" (UID: "f4314f51-8cae-40ef-a986-942812e3210e"). InnerVolumeSpecName "kube-api-access-l8dsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:32.003010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:32.002979    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4314f51-8cae-40ef-a986-942812e3210e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:32.003010 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:32.003006    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8dsl\" (UniqueName: \"kubernetes.io/projected/f4314f51-8cae-40ef-a986-942812e3210e-kube-api-access-l8dsl\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:32.665905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:32.665867    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85" event={"ID":"f4314f51-8cae-40ef-a986-942812e3210e","Type":"ContainerDied","Data":"c1445113d9c50325cfe153199703cfe7dd14b8721ef4ab7fdc877b27c24ebe44"}
Oct 31 02:05:32.666083 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:32.665927    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1445113d9c50325cfe153199703cfe7dd14b8721ef4ab7fdc877b27c24ebe44"
Oct 31 02:05:32.666083 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:32.665879    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"
Oct 31 02:05:40.083460 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:40.083427    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"]
Oct 31 02:05:40.083916 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:40.083654    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" containerID="cri-o://82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd" gracePeriod=30
Oct 31 02:05:41.766386 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:41.766348    5419 scope.go:122] "RemoveContainer" containerID="30a61eb097092afdb92a2004a90aabe791982381354f7cd4551ebeb035e631aa"
Oct 31 02:05:43.620879 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:43.620840    5419 patch_prober.go:29] interesting pod/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:05:43.621426 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:43.620923    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:05:43.979100 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:43.979063    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"]
Oct 31 02:05:43.979339 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:43.979318    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy" containerID="cri-o://6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25" gracePeriod=30
Oct 31 02:05:44.028268 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:44.028230    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"]
Oct 31 02:05:44.028489 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:44.028447    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy" containerID="cri-o://78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580" gracePeriod=30
Oct 31 02:05:44.585426 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:05:44.585393    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:05:45.344713 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.344650    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 02:05:45.376520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376481    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376541    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376663    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376693    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376898 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376730    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376898 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376768    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376898 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376796    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376898 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376836    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.376898 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376874    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data\") pod \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\" (UID: \"7fbc5c62-f0c9-4831-8fe3-4e429de2a943\") "
Oct 31 02:05:45.377206 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.376991    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:45.377206 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377015    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:45.377206 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377076    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:45.377408 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377211    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:45.377408 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377302    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:45.377740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377683    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.377740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377712    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.377740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377727    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.377740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377740    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.377740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.377754    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.379027 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.379000    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:45.379185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.379031    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "kube-api-access-7mrmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:45.379185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.379060    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:05:45.379387 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.379373    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token" pod "7fbc5c62-f0c9-4831-8fe3-4e429de2a943" (UID: "7fbc5c62-f0c9-4831-8fe3-4e429de2a943"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:45.478994 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.478961    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrmd\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-kube-api-access-7mrmd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.478994 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.478993    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.478994 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.479003    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.479253 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.479012    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fbc5c62-f0c9-4831-8fe3-4e429de2a943-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:45.761100 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.761066    5419 generic.go:356] "Generic (PLEG): container finished" podID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerID="82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd" exitCode=0
Oct 31 02:05:45.761309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.761118    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"
Oct 31 02:05:45.761309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.761163    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" event={"ID":"7fbc5c62-f0c9-4831-8fe3-4e429de2a943","Type":"ContainerDied","Data":"82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd"}
Oct 31 02:05:45.761309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.761224    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r" event={"ID":"7fbc5c62-f0c9-4831-8fe3-4e429de2a943","Type":"ContainerDied","Data":"ffcc153bda14436fc01e22dd797295241719af369bef77244bb21cfc40aa54d4"}
Oct 31 02:05:45.761309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.761253    5419 scope.go:122] "RemoveContainer" containerID="82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd"
Oct 31 02:05:45.772567 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.772546    5419 scope.go:122] "RemoveContainer" containerID="82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd"
Oct 31 02:05:45.772903 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:05:45.772881    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd\": container with ID starting with 82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd not found: ID does not exist" containerID="82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd"
Oct 31 02:05:45.772975 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.772914    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd"} err="failed to get container status \"82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd\": rpc error: code = NotFound desc = could not find container \"82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd\": container with ID starting with 82aa8aa92f15756def67063d9c6e7826fb7b60f7b1f75e132c02aa581e1ecdfd not found: ID does not exist"
Oct 31 02:05:45.927026 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.926991    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"]
Oct 31 02:05:45.930299 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:45.930271    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057e2d-openshift-default-64f8fbc7cf-q888r"]
Oct 31 02:05:46.837688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:46.837660    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" path="/var/lib/kubelet/pods/7fbc5c62-f0c9-4831-8fe3-4e429de2a943/volumes"
Oct 31 02:05:49.290447 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.290416    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 02:05:49.321032 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.321006    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 02:05:49.324254 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324235    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324341 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324321    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324385 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324354    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324421 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324392    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324457 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324434    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324514    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.324603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324537    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324565    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324718 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324617    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324718 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324658    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy\") pod \"f2a8db1b-fdcc-469d-88bc-1172e8860382\" (UID: \"f2a8db1b-fdcc-469d-88bc-1172e8860382\") "
Oct 31 02:05:49.324718 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324615    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.324879 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324767    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.324879 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324804    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.324879 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.324826    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:49.325237 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.325216    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.325346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.325242    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.325346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.325254    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.325346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.325262    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a8db1b-fdcc-469d-88bc-1172e8860382-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.325346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.325272    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.326880 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.326852    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.327073 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.327051    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "kube-api-access-nvtx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:49.327183 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.327159    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:05:49.327235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.327183    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token" pod "f2a8db1b-fdcc-469d-88bc-1172e8860382" (UID: "f2a8db1b-fdcc-469d-88bc-1172e8860382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:49.425855 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.425830    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426032 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.425880    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426032 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.425944    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426032 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.425974    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426032 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426016    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426044    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426099    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426204    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426277    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.426307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426300    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.426572 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426311    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:05:49.426572 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426342    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t\") pod \"defd1519-70d7-4a57-8f72-f6a32b7e670b\" (UID: \"defd1519-70d7-4a57-8f72-f6a32b7e670b\") "
Oct 31 02:05:49.426572 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426494    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.426572 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426507    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.426837 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426817    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.426912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426845    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtx4\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-kube-api-access-nvtx4\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.426912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426862    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defd1519-70d7-4a57-8f72-f6a32b7e670b-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.426912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426876    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.426912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426889    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.426912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426902    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.427302 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426917    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.427302 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426931    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a8db1b-fdcc-469d-88bc-1172e8860382-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.427302 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.426944    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.428407 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.428388    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:49.428484 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.428435    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:05:49.428975 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.428953    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:05:49.428975 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.428966    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t" pod "defd1519-70d7-4a57-8f72-f6a32b7e670b" (UID: "defd1519-70d7-4a57-8f72-f6a32b7e670b"). InnerVolumeSpecName "kube-api-access-6gc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:05:49.527719 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.527643    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gc9t\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-kube-api-access-6gc9t\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.527719 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.527670    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.527719 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.527680    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.527719 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.527688    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defd1519-70d7-4a57-8f72-f6a32b7e670b-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:05:49.790545 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.790460    5419 generic.go:356] "Generic (PLEG): container finished" podID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerID="78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580" exitCode=0
Oct 31 02:05:49.790545 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.790511    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"
Oct 31 02:05:49.790775 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.790542    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" event={"ID":"defd1519-70d7-4a57-8f72-f6a32b7e670b","Type":"ContainerDied","Data":"78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580"}
Oct 31 02:05:49.790775 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.790601    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b" event={"ID":"defd1519-70d7-4a57-8f72-f6a32b7e670b","Type":"ContainerDied","Data":"0124212f546c228ff2d9f30862d9eb59d78f7e6fc857ea82f134f9f63e658836"}
Oct 31 02:05:49.790775 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.790628    5419 scope.go:122] "RemoveContainer" containerID="78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580"
Oct 31 02:05:49.792034 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.792010    5419 generic.go:356] "Generic (PLEG): container finished" podID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerID="6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25" exitCode=0
Oct 31 02:05:49.792197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.792065    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"
Oct 31 02:05:49.792197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.792092    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" event={"ID":"f2a8db1b-fdcc-469d-88bc-1172e8860382","Type":"ContainerDied","Data":"6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25"}
Oct 31 02:05:49.792197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.792169    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l" event={"ID":"f2a8db1b-fdcc-469d-88bc-1172e8860382","Type":"ContainerDied","Data":"7f62ea6a7ec73565e5f8d18a3278dd9f5ff7985e9a6b9835e314834888bd721a"}
Oct 31 02:05:49.802903 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.802887    5419 scope.go:122] "RemoveContainer" containerID="78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580"
Oct 31 02:05:49.803196 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:05:49.803179    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580\": container with ID starting with 78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580 not found: ID does not exist" containerID="78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580"
Oct 31 02:05:49.803256 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.803204    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580"} err="failed to get container status \"78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580\": rpc error: code = NotFound desc = could not find container \"78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580\": container with ID starting with 78ea1d23c4953d60d1252fc3700e227bf78b5ed1c3cad53a4a00bb9f47bc8580 not found: ID does not exist"
Oct 31 02:05:49.803256 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.803220    5419 scope.go:122] "RemoveContainer" containerID="6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25"
Oct 31 02:05:49.813046 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.813028    5419 scope.go:122] "RemoveContainer" containerID="6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25"
Oct 31 02:05:49.813341 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:05:49.813323    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25\": container with ID starting with 6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25 not found: ID does not exist" containerID="6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25"
Oct 31 02:05:49.813420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.813349    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25"} err="failed to get container status \"6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25\": rpc error: code = NotFound desc = could not find container \"6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25\": container with ID starting with 6cd64792150e5c237d6cffe29eda810c83769ff15ef82d0f1449860050fa0a25 not found: ID does not exist"
Oct 31 02:05:49.954646 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.954610    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"]
Oct 31 02:05:49.958125 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:49.958101    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-28f4l"]
Oct 31 02:05:50.066556 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:50.066468    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"]
Oct 31 02:05:50.069806 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:50.069777    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-00e840-openshift-default-59b96fb879-2xg4b"]
Oct 31 02:05:50.837079 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:50.837047    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" path="/var/lib/kubelet/pods/defd1519-70d7-4a57-8f72-f6a32b7e670b/volumes"
Oct 31 02:05:50.837506 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:05:50.837492    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" path="/var/lib/kubelet/pods/f2a8db1b-fdcc-469d-88bc-1172e8860382/volumes"
Oct 31 02:06:13.054975 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:13.054938    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"]
Oct 31 02:06:13.055554 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:13.055158    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy" containerID="cri-o://d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503" gracePeriod=30
Oct 31 02:06:14.583547 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:06:14.583510    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:06:18.315782 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.315757    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:06:18.403417 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403320    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403417 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403404    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403687 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403437    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403687 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403488    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403687 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403564    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403687 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403620    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403687 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403663    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403696    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403768    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs\") pod \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\" (UID: \"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0\") "
Oct 31 02:06:18.403980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403796    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:18.403980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.403882    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404030    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404221    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404250    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404402    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404419    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404433    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404447    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.404464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.404460    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.405967 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.405942    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:18.406112 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.406085    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:18.406376 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.406355    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:06:18.406581 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.406564    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf" pod "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" (UID: "4db42ddf-b07d-4263-bc7a-fd9ec331f8a0"). InnerVolumeSpecName "kube-api-access-w8ktf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:18.505195 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.505164    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8ktf\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-kube-api-access-w8ktf\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.505195 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.505195    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.505401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.505208    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:18.505401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:18.505220    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:19.032696 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.032663    5419 generic.go:356] "Generic (PLEG): container finished" podID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerID="d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503" exitCode=0
Oct 31 02:06:19.032893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.032728    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"
Oct 31 02:06:19.032893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.032742    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" event={"ID":"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0","Type":"ContainerDied","Data":"d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503"}
Oct 31 02:06:19.032893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.032807    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf" event={"ID":"4db42ddf-b07d-4263-bc7a-fd9ec331f8a0","Type":"ContainerDied","Data":"3bec60159c4bef164de5622b19ee90ccece5ca5b0136febdc4da6c0f2f1ab636"}
Oct 31 02:06:19.032893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.032839    5419 scope.go:122] "RemoveContainer" containerID="d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503"
Oct 31 02:06:19.042828 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.042804    5419 scope.go:122] "RemoveContainer" containerID="d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503"
Oct 31 02:06:19.043108 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:06:19.043087    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503\": container with ID starting with d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503 not found: ID does not exist" containerID="d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503"
Oct 31 02:06:19.043215 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.043117    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503"} err="failed to get container status \"d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503\": rpc error: code = NotFound desc = could not find container \"d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503\": container with ID starting with d14b9fa3c9a2132b96b018b23d7602a18ac12a49fa420c502f2fb33587e96503 not found: ID does not exist"
Oct 31 02:06:19.246513 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.246474    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"]
Oct 31 02:06:19.296179 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:19.296133    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8f6b5b-openshift-default-696c4598cf-7rblf"]
Oct 31 02:06:20.838509 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:20.838479    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" path="/var/lib/kubelet/pods/4db42ddf-b07d-4263-bc7a-fd9ec331f8a0/volumes"
Oct 31 02:06:22.735527 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:22.735492    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:06:29.328116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328077    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"]
Oct 31 02:06:29.328708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328692    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy"
Oct 31 02:06:29.328759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328709    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy"
Oct 31 02:06:29.328759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328727    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy"
Oct 31 02:06:29.328759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328732    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy"
Oct 31 02:06:29.328759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328745    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f4314f51-8cae-40ef-a986-942812e3210e" containerName="revoke-keys"
Oct 31 02:06:29.328759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328750    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4314f51-8cae-40ef-a986-942812e3210e" containerName="revoke-keys"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328773    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328778    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328793    5419 cpu_manager.go:414] "containerMap: removing container" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328797    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328891    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbc5c62-f0c9-4831-8fe3-4e429de2a943" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328900    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="defd1519-70d7-4a57-8f72-f6a32b7e670b" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328909    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a8db1b-fdcc-469d-88bc-1172e8860382" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328914    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db42ddf-b07d-4263-bc7a-fd9ec331f8a0" containerName="istio-proxy"
Oct 31 02:06:29.328921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.328922    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4314f51-8cae-40ef-a986-942812e3210e" containerName="revoke-keys"
Oct 31 02:06:29.333887 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.333872    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.386277 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.386248    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:06:29.386447 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.386371    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:06:29.436209 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.435926    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:06:29.436209 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.436086    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:06:29.440651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.439944    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.440651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.439988    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz2nz\" (UniqueName: \"kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.440651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.440123    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.440651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.440385    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"]
Oct 31 02:06:29.542189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.542125    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.542414 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.542309    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.542414 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.542346    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz2nz\" (UniqueName: \"kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.542931 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.542902    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.545188 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.545165    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.598787 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.598717    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz2nz\" (UniqueName: \"kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz\") pod \"maas-api-revoke-sub-23c7b04c-gkqgh\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:29.647709 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:29.647683    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:30.023047 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:30.023014    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"]
Oct 31 02:06:30.024376 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:06:30.024348    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7fd4ca_ab96_4ddd_803b_782f50ef6166.slice/crio-f05735c2abdcb58239e9aea472340da9a61806454c7127992e34ed3506a7d8ef WatchSource:0}: Error finding container f05735c2abdcb58239e9aea472340da9a61806454c7127992e34ed3506a7d8ef: Status 404 returned error can't find the container with id f05735c2abdcb58239e9aea472340da9a61806454c7127992e34ed3506a7d8ef
Oct 31 02:06:30.108721 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:30.108687    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh" event={"ID":"9c7fd4ca-ab96-4ddd-803b-782f50ef6166","Type":"ContainerStarted","Data":"f05735c2abdcb58239e9aea472340da9a61806454c7127992e34ed3506a7d8ef"}
Oct 31 02:06:31.151219 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:31.147989    5419 generic.go:356] "Generic (PLEG): container finished" podID="9c7fd4ca-ab96-4ddd-803b-782f50ef6166" containerID="8870ab99f5603f50dfdd3b583350a9f7d95ac39bac411fdccfe78857782da0f0" exitCode=0
Oct 31 02:06:31.151219 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:31.148202    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh" event={"ID":"9c7fd4ca-ab96-4ddd-803b-782f50ef6166","Type":"ContainerDied","Data":"8870ab99f5603f50dfdd3b583350a9f7d95ac39bac411fdccfe78857782da0f0"}
Oct 31 02:06:31.228072 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:31.228037    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"]
Oct 31 02:06:31.228349 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:31.228322    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" podUID="c7da9ad5-4039-429b-96cc-14f051bda03d" containerName="istio-proxy" containerID="cri-o://9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740" gracePeriod=30
Oct 31 02:06:32.317025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.316998    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:32.494956 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.494926    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca\") pod \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") "
Oct 31 02:06:32.495154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.495078    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca\") pod \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") "
Oct 31 02:06:32.495154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.495119    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz\" (UniqueName: \"kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz\") pod \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\" (UID: \"9c7fd4ca-ab96-4ddd-803b-782f50ef6166\") "
Oct 31 02:06:32.495338 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.495313    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca" pod "9c7fd4ca-ab96-4ddd-803b-782f50ef6166" (UID: "9c7fd4ca-ab96-4ddd-803b-782f50ef6166"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:32.495563 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.495543    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:32.497449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.497427    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca" pod "9c7fd4ca-ab96-4ddd-803b-782f50ef6166" (UID: "9c7fd4ca-ab96-4ddd-803b-782f50ef6166"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:32.497449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.497441    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz" pod "9c7fd4ca-ab96-4ddd-803b-782f50ef6166" (UID: "9c7fd4ca-ab96-4ddd-803b-782f50ef6166"). InnerVolumeSpecName "kube-api-access-bz2nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:32.596518 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.596483    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:32.596518 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:32.596511    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz2nz\" (UniqueName: \"kubernetes.io/projected/9c7fd4ca-ab96-4ddd-803b-782f50ef6166-kube-api-access-bz2nz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:33.165393 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:33.165317    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"
Oct 31 02:06:33.165393 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:33.165344    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh" event={"ID":"9c7fd4ca-ab96-4ddd-803b-782f50ef6166","Type":"ContainerDied","Data":"f05735c2abdcb58239e9aea472340da9a61806454c7127992e34ed3506a7d8ef"}
Oct 31 02:06:33.165393 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:33.165391    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f05735c2abdcb58239e9aea472340da9a61806454c7127992e34ed3506a7d8ef"
Oct 31 02:06:33.566845 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:33.566809    5419 patch_prober.go:29] interesting pod/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:06:33.567279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:33.566862    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" podUID="c7da9ad5-4039-429b-96cc-14f051bda03d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:06:36.504750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.504697    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:06:36.538905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.538879    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.538933    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.538982    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539023    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539076    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539107    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539216    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:36.539266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539225    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539485 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539324    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539485 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539430    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket\") pod \"c7da9ad5-4039-429b-96cc-14f051bda03d\" (UID: \"c7da9ad5-4039-429b-96cc-14f051bda03d\") "
Oct 31 02:06:36.539485 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539428    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:36.539651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539631    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:36.539699 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539677    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:36.539745 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.539725    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:36.540197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.540175    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.540286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.540205    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.540286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.540220    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.540286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.540236    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7da9ad5-4039-429b-96cc-14f051bda03d-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.540286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.540250    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.541454 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.541426    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:06:36.541590 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.541574    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:36.541934 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.541904    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:06:36.541999 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.541941    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp" pod "c7da9ad5-4039-429b-96cc-14f051bda03d" (UID: "c7da9ad5-4039-429b-96cc-14f051bda03d"). InnerVolumeSpecName "kube-api-access-wkftp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:36.641123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.641081    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.641123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.641113    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.641123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.641127    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkftp\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-kube-api-access-wkftp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:36.641123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:36.641159    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7da9ad5-4039-429b-96cc-14f051bda03d-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:37.206369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.206285    5419 generic.go:356] "Generic (PLEG): container finished" podID="c7da9ad5-4039-429b-96cc-14f051bda03d" containerID="9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740" exitCode=0
Oct 31 02:06:37.206369 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.206336    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"
Oct 31 02:06:37.206555 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.206324    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" event={"ID":"c7da9ad5-4039-429b-96cc-14f051bda03d","Type":"ContainerDied","Data":"9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740"}
Oct 31 02:06:37.206555 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.206454    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72" event={"ID":"c7da9ad5-4039-429b-96cc-14f051bda03d","Type":"ContainerDied","Data":"824ea09a4c17e46163420e5d03eef9de14b943b47c35e248dfef384497f78966"}
Oct 31 02:06:37.206555 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.206477    5419 scope.go:122] "RemoveContainer" containerID="9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740"
Oct 31 02:06:37.216712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.216695    5419 scope.go:122] "RemoveContainer" containerID="9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740"
Oct 31 02:06:37.216958 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:06:37.216942    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740\": container with ID starting with 9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740 not found: ID does not exist" containerID="9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740"
Oct 31 02:06:37.217017 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.216975    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740"} err="failed to get container status \"9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740\": rpc error: code = NotFound desc = could not find container \"9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740\": container with ID starting with 9e11ac79f799e145e376aff01d796ffdf64780b2e8ebdfeb0f7bbf25eaca9740 not found: ID does not exist"
Oct 31 02:06:37.370263 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.370229    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"]
Oct 31 02:06:37.419704 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.419675    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8114c9-openshift-default-5c444cfb5-xmg72"]
Oct 31 02:06:37.787745 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.787710    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"]
Oct 31 02:06:37.788294 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.788279    5419 cpu_manager.go:414] "containerMap: removing container" podUID="9c7fd4ca-ab96-4ddd-803b-782f50ef6166" containerName="revoke-keys"
Oct 31 02:06:37.788347 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.788295    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7fd4ca-ab96-4ddd-803b-782f50ef6166" containerName="revoke-keys"
Oct 31 02:06:37.788347 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.788322    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c7da9ad5-4039-429b-96cc-14f051bda03d" containerName="istio-proxy"
Oct 31 02:06:37.788347 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.788328    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7da9ad5-4039-429b-96cc-14f051bda03d" containerName="istio-proxy"
Oct 31 02:06:37.788452 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.788420    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7fd4ca-ab96-4ddd-803b-782f50ef6166" containerName="revoke-keys"
Oct 31 02:06:37.788452 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.788431    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7da9ad5-4039-429b-96cc-14f051bda03d" containerName="istio-proxy"
Oct 31 02:06:37.793063 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.793040    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.853376 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.853275    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.860302    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.860457    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.860575    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.870406    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.870454    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.870485    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtxg\" (UniqueName: \"kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.871835 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.871727    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"]
Oct 31 02:06:37.972078 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.972045    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.972321 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.972083    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.972321 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.972105    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtxg\" (UniqueName: \"kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.972788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.972762    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:37.974831 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:37.974806    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:38.035057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:38.028929    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtxg\" (UniqueName: \"kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:38.107200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:38.107118    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:38.484531 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:38.484502    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"]
Oct 31 02:06:38.487191 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:06:38.487164    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a17e423_8bec_41e6_9f20_21654846c7ca.slice/crio-da4dde72aa04668e6d9a9951dd295e6f7f85740e18840d8cf22a6b8667fd4e89 WatchSource:0}: Error finding container da4dde72aa04668e6d9a9951dd295e6f7f85740e18840d8cf22a6b8667fd4e89: Status 404 returned error can't find the container with id da4dde72aa04668e6d9a9951dd295e6f7f85740e18840d8cf22a6b8667fd4e89
Oct 31 02:06:38.844380 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:38.842591    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7da9ad5-4039-429b-96cc-14f051bda03d" path="/var/lib/kubelet/pods/c7da9ad5-4039-429b-96cc-14f051bda03d/volumes"
Oct 31 02:06:39.227893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:39.227860    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd" event={"ID":"6a17e423-8bec-41e6-9f20-21654846c7ca","Type":"ContainerStarted","Data":"d0093d1d68e69c728d5114529138b7fdce50817bf867c50002f1c911d804ce1c"}
Oct 31 02:06:39.228081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:39.227915    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd" event={"ID":"6a17e423-8bec-41e6-9f20-21654846c7ca","Type":"ContainerStarted","Data":"da4dde72aa04668e6d9a9951dd295e6f7f85740e18840d8cf22a6b8667fd4e89"}
Oct 31 02:06:39.343910 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:39.343855    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd" podStartSLOduration=2.343836412 podStartE2EDuration="2.343836412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:06:39.341381378 +0000 UTC m=+3720.642337955" watchObservedRunningTime="2026-10-31 02:06:39.343836412 +0000 UTC m=+3720.644792989"
Oct 31 02:06:40.115645 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.115616    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"]
Oct 31 02:06:40.119712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.119696    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.210057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.210019    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhzg\" (UniqueName: \"kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.210275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.210065    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.210275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.210183    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.224947 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.224919    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"]
Oct 31 02:06:40.235839 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.235809    5419 generic.go:356] "Generic (PLEG): container finished" podID="6a17e423-8bec-41e6-9f20-21654846c7ca" containerID="d0093d1d68e69c728d5114529138b7fdce50817bf867c50002f1c911d804ce1c" exitCode=0
Oct 31 02:06:40.235981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.235891    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd" event={"ID":"6a17e423-8bec-41e6-9f20-21654846c7ca","Type":"ContainerDied","Data":"d0093d1d68e69c728d5114529138b7fdce50817bf867c50002f1c911d804ce1c"}
Oct 31 02:06:40.311514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.311473    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhzg\" (UniqueName: \"kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.311708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.311560    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.311708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.311649    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.312440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.312343    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.314194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.314174    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.367852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.367785    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhzg\" (UniqueName: \"kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg\") pod \"maas-api-revoke-sub-a83a034f-p6gkr\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.432947 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.432920    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:40.808430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:40.808406    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"]
Oct 31 02:06:40.810061 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:06:40.810018    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99941559_bf52_4b0d_a930_e7f9282007ef.slice/crio-a67bd64ee5dfdb525bdf8093f726726992af1ae0596cf8599fe024a6e0426baa WatchSource:0}: Error finding container a67bd64ee5dfdb525bdf8093f726726992af1ae0596cf8599fe024a6e0426baa: Status 404 returned error can't find the container with id a67bd64ee5dfdb525bdf8093f726726992af1ae0596cf8599fe024a6e0426baa
Oct 31 02:06:41.244585 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.244549    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr" event={"ID":"99941559-bf52-4b0d-a930-e7f9282007ef","Type":"ContainerStarted","Data":"9e71316acfb6c0ca39bf1bde63b82c5464b4b947ba21b7c27ad5f1c6a949494a"}
Oct 31 02:06:41.245052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.244627    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr" event={"ID":"99941559-bf52-4b0d-a930-e7f9282007ef","Type":"ContainerStarted","Data":"a67bd64ee5dfdb525bdf8093f726726992af1ae0596cf8599fe024a6e0426baa"}
Oct 31 02:06:41.359165 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.359041    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr" podStartSLOduration=1.359026403 podStartE2EDuration="1.359026403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:06:41.356568525 +0000 UTC m=+3722.657525127" watchObservedRunningTime="2026-10-31 02:06:41.359026403 +0000 UTC m=+3722.659982981"
Oct 31 02:06:41.389239 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.389217    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:41.532893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.532861    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca\") pod \"6a17e423-8bec-41e6-9f20-21654846c7ca\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") "
Oct 31 02:06:41.533081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.532937    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca\") pod \"6a17e423-8bec-41e6-9f20-21654846c7ca\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") "
Oct 31 02:06:41.533081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.532976    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg\" (UniqueName: \"kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg\") pod \"6a17e423-8bec-41e6-9f20-21654846c7ca\" (UID: \"6a17e423-8bec-41e6-9f20-21654846c7ca\") "
Oct 31 02:06:41.533318 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.533289    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca" pod "6a17e423-8bec-41e6-9f20-21654846c7ca" (UID: "6a17e423-8bec-41e6-9f20-21654846c7ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:41.533609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.533585    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:41.535227 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.535197    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca" pod "6a17e423-8bec-41e6-9f20-21654846c7ca" (UID: "6a17e423-8bec-41e6-9f20-21654846c7ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:41.535339 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.535283    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg" pod "6a17e423-8bec-41e6-9f20-21654846c7ca" (UID: "6a17e423-8bec-41e6-9f20-21654846c7ca"). InnerVolumeSpecName "kube-api-access-6xtxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:41.635030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.634957    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a17e423-8bec-41e6-9f20-21654846c7ca-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:41.635030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.634986    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xtxg\" (UniqueName: \"kubernetes.io/projected/6a17e423-8bec-41e6-9f20-21654846c7ca-kube-api-access-6xtxg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:41.808803 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:41.808776    5419 scope.go:122] "RemoveContainer" containerID="c0740d0e5f747aa3dc8a60823a898205b7390da919f318a2187cc23d9e8d7218"
Oct 31 02:06:42.253066 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:42.253030    5419 generic.go:356] "Generic (PLEG): container finished" podID="99941559-bf52-4b0d-a930-e7f9282007ef" containerID="9e71316acfb6c0ca39bf1bde63b82c5464b4b947ba21b7c27ad5f1c6a949494a" exitCode=0
Oct 31 02:06:42.253556 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:42.253114    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr" event={"ID":"99941559-bf52-4b0d-a930-e7f9282007ef","Type":"ContainerDied","Data":"9e71316acfb6c0ca39bf1bde63b82c5464b4b947ba21b7c27ad5f1c6a949494a"}
Oct 31 02:06:42.254646 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:42.254620    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd" event={"ID":"6a17e423-8bec-41e6-9f20-21654846c7ca","Type":"ContainerDied","Data":"da4dde72aa04668e6d9a9951dd295e6f7f85740e18840d8cf22a6b8667fd4e89"}
Oct 31 02:06:42.254773 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:42.254648    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"
Oct 31 02:06:42.254773 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:42.254664    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4dde72aa04668e6d9a9951dd295e6f7f85740e18840d8cf22a6b8667fd4e89"
Oct 31 02:06:43.415321 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.415294    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:43.559021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.558936    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca\") pod \"99941559-bf52-4b0d-a930-e7f9282007ef\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") "
Oct 31 02:06:43.559218 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.559036    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca\") pod \"99941559-bf52-4b0d-a930-e7f9282007ef\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") "
Oct 31 02:06:43.559218 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.559175    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg\" (UniqueName: \"kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg\") pod \"99941559-bf52-4b0d-a930-e7f9282007ef\" (UID: \"99941559-bf52-4b0d-a930-e7f9282007ef\") "
Oct 31 02:06:43.559458 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.559432    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca" pod "99941559-bf52-4b0d-a930-e7f9282007ef" (UID: "99941559-bf52-4b0d-a930-e7f9282007ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:43.559806 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.559776    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:43.561329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.561304    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca" pod "99941559-bf52-4b0d-a930-e7f9282007ef" (UID: "99941559-bf52-4b0d-a930-e7f9282007ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:06:43.561995 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.561966    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg" pod "99941559-bf52-4b0d-a930-e7f9282007ef" (UID: "99941559-bf52-4b0d-a930-e7f9282007ef"). InnerVolumeSpecName "kube-api-access-phhzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:43.661390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.661357    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99941559-bf52-4b0d-a930-e7f9282007ef-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:43.661390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:43.661386    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhzg\" (UniqueName: \"kubernetes.io/projected/99941559-bf52-4b0d-a930-e7f9282007ef-kube-api-access-phhzg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:44.268419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:44.268394    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"
Oct 31 02:06:44.268419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:44.268411    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr" event={"ID":"99941559-bf52-4b0d-a930-e7f9282007ef","Type":"ContainerDied","Data":"a67bd64ee5dfdb525bdf8093f726726992af1ae0596cf8599fe024a6e0426baa"}
Oct 31 02:06:44.268652 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:44.268455    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a67bd64ee5dfdb525bdf8093f726726992af1ae0596cf8599fe024a6e0426baa"
Oct 31 02:06:44.582643 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:06:44.582549    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:06:49.367875 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:49.367840    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"]
Oct 31 02:06:54.368296 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.368253    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq" podUID="7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" containerName="payload-pre-processing" containerID="cri-o://05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df" gracePeriod=25
Oct 31 02:06:54.628753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.628685    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 02:06:54.697486 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.697455    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume\") pod \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") "
Oct 31 02:06:54.697670 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.697550    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5\" (UniqueName: \"kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5\") pod \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\" (UID: \"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d\") "
Oct 31 02:06:54.697907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.697878    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume" pod "7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" (UID: "7abbc023-4e14-45e3-a2ca-2ca0e4698d0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:06:54.698175 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.698150    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:54.699870 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.699845    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5" pod "7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" (UID: "7abbc023-4e14-45e3-a2ca-2ca0e4698d0d"). InnerVolumeSpecName "kube-api-access-njzm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:06:54.799382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:54.799345    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njzm5\" (UniqueName: \"kubernetes.io/projected/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d-kube-api-access-njzm5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:06:55.348771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.348729    5419 generic.go:356] "Generic (PLEG): container finished" podID="7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" containerID="05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df" exitCode=0
Oct 31 02:06:55.349000 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.348777    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"
Oct 31 02:06:55.349000 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.348817    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq" event={"ID":"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d","Type":"ContainerDied","Data":"05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df"}
Oct 31 02:06:55.349000 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.348877    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq" event={"ID":"7abbc023-4e14-45e3-a2ca-2ca0e4698d0d","Type":"ContainerDied","Data":"4589e7b89e7d5eb51d53c75a957131bd2495f29f734035c77878af8ee93452e5"}
Oct 31 02:06:55.349000 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.348903    5419 scope.go:122] "RemoveContainer" containerID="05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df"
Oct 31 02:06:55.359251 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.359228    5419 scope.go:122] "RemoveContainer" containerID="05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df"
Oct 31 02:06:55.359503 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:06:55.359484    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df\": container with ID starting with 05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df not found: ID does not exist" containerID="05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df"
Oct 31 02:06:55.359554 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.359510    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df"} err="failed to get container status \"05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df\": rpc error: code = NotFound desc = could not find container \"05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df\": container with ID starting with 05b4b81cb6d5ca28eca7f5a329578053f97cb98cde42204892529d6714ab52df not found: ID does not exist"
Oct 31 02:06:55.511893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.511861    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"]
Oct 31 02:06:55.561127 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:55.561095    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-98b70c-5b8dd684bc-vx5pq"]
Oct 31 02:06:56.845916 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:06:56.845884    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" path="/var/lib/kubelet/pods/7abbc023-4e14-45e3-a2ca-2ca0e4698d0d/volumes"
Oct 31 02:07:12.527461 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:12.527429    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"]
Oct 31 02:07:13.077825 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.077794    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"]
Oct 31 02:07:13.078360 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078347    5419 cpu_manager.go:414] "containerMap: removing container" podUID="99941559-bf52-4b0d-a930-e7f9282007ef" containerName="revoke-keys"
Oct 31 02:07:13.078412 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078362    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99941559-bf52-4b0d-a930-e7f9282007ef" containerName="revoke-keys"
Oct 31 02:07:13.078412 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078389    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" containerName="payload-pre-processing"
Oct 31 02:07:13.078412 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078394    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" containerName="payload-pre-processing"
Oct 31 02:07:13.078412 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078403    5419 cpu_manager.go:414] "containerMap: removing container" podUID="6a17e423-8bec-41e6-9f20-21654846c7ca" containerName="revoke-keys"
Oct 31 02:07:13.078412 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078408    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a17e423-8bec-41e6-9f20-21654846c7ca" containerName="revoke-keys"
Oct 31 02:07:13.078580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078504    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="99941559-bf52-4b0d-a930-e7f9282007ef" containerName="revoke-keys"
Oct 31 02:07:13.078580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078515    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7abbc023-4e14-45e3-a2ca-2ca0e4698d0d" containerName="payload-pre-processing"
Oct 31 02:07:13.078580 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.078523    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a17e423-8bec-41e6-9f20-21654846c7ca" containerName="revoke-keys"
Oct 31 02:07:13.081706 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.081691    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.139508 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.139467    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:07:13.139773 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.139487    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:13.142933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.142902    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"]
Oct 31 02:07:13.189024 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.188996    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:07:13.189024 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.189018    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:07:13.221168 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.220544    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.221168 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.220709    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8mn\" (UniqueName: \"kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.221168 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.220753    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.322458 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.322421    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.322615 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.322537    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8mn\" (UniqueName: \"kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.322615 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.322562    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.323248 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.323229    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.324906 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.324881    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.378913 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.378844    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8mn\" (UniqueName: \"kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.394747 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.394722    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:13.771906 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:13.771877    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"]
Oct 31 02:07:13.773412 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:13.773377    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02629c88_18e5_44a5_982d_a0240a62b90a.slice/crio-ca9efd32dda730d7cdef8f0334fdb0aa690b94d658b3fc607799f1a69ea14d32 WatchSource:0}: Error finding container ca9efd32dda730d7cdef8f0334fdb0aa690b94d658b3fc607799f1a69ea14d32: Status 404 returned error can't find the container with id ca9efd32dda730d7cdef8f0334fdb0aa690b94d658b3fc607799f1a69ea14d32
Oct 31 02:07:14.484437 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:14.484402    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w" event={"ID":"02629c88-18e5-44a5-982d-a0240a62b90a","Type":"ContainerStarted","Data":"b37d7c768ceb495db2dd4003fc2bfbcb220a4c6c7499c516ddf03d5654c6a55b"}
Oct 31 02:07:14.484630 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:14.484470    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w" event={"ID":"02629c88-18e5-44a5-982d-a0240a62b90a","Type":"ContainerStarted","Data":"ca9efd32dda730d7cdef8f0334fdb0aa690b94d658b3fc607799f1a69ea14d32"}
Oct 31 02:07:14.507596 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:14.507570    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"]
Oct 31 02:07:14.557460 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:14.557434    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-98b70c-s26rd"]
Oct 31 02:07:14.582515 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:14.582480    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:07:14.616829 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:14.616770    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w" podStartSLOduration=1.616747417 podStartE2EDuration="1.616747417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:14.616458341 +0000 UTC m=+3755.917414940" watchObservedRunningTime="2026-10-31 02:07:14.616747417 +0000 UTC m=+3755.917703996"
Oct 31 02:07:14.837482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:14.837444    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a17e423-8bec-41e6-9f20-21654846c7ca" path="/var/lib/kubelet/pods/6a17e423-8bec-41e6-9f20-21654846c7ca/volumes"
Oct 31 02:07:15.493256 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:15.493221    5419 generic.go:356] "Generic (PLEG): container finished" podID="02629c88-18e5-44a5-982d-a0240a62b90a" containerID="b37d7c768ceb495db2dd4003fc2bfbcb220a4c6c7499c516ddf03d5654c6a55b" exitCode=0
Oct 31 02:07:15.493467 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:15.493307    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w" event={"ID":"02629c88-18e5-44a5-982d-a0240a62b90a","Type":"ContainerDied","Data":"b37d7c768ceb495db2dd4003fc2bfbcb220a4c6c7499c516ddf03d5654c6a55b"}
Oct 31 02:07:16.567200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.567164    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:07:16.663300 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.663272    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:16.766768 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.766666    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca\") pod \"02629c88-18e5-44a5-982d-a0240a62b90a\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") "
Oct 31 02:07:16.767562 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.767530    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca\") pod \"02629c88-18e5-44a5-982d-a0240a62b90a\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") "
Oct 31 02:07:16.768401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.768380    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn\" (UniqueName: \"kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn\") pod \"02629c88-18e5-44a5-982d-a0240a62b90a\" (UID: \"02629c88-18e5-44a5-982d-a0240a62b90a\") "
Oct 31 02:07:16.769267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.767402    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca" pod "02629c88-18e5-44a5-982d-a0240a62b90a" (UID: "02629c88-18e5-44a5-982d-a0240a62b90a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:16.769590 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.769567    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:16.774823 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.774789    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca" pod "02629c88-18e5-44a5-982d-a0240a62b90a" (UID: "02629c88-18e5-44a5-982d-a0240a62b90a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:16.790735 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.790701    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn" pod "02629c88-18e5-44a5-982d-a0240a62b90a" (UID: "02629c88-18e5-44a5-982d-a0240a62b90a"). InnerVolumeSpecName "kube-api-access-5g8mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:16.871327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.871287    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02629c88-18e5-44a5-982d-a0240a62b90a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:16.871327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:16.871329    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g8mn\" (UniqueName: \"kubernetes.io/projected/02629c88-18e5-44a5-982d-a0240a62b90a-kube-api-access-5g8mn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:17.530167 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:17.518451    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w" event={"ID":"02629c88-18e5-44a5-982d-a0240a62b90a","Type":"ContainerDied","Data":"ca9efd32dda730d7cdef8f0334fdb0aa690b94d658b3fc607799f1a69ea14d32"}
Oct 31 02:07:17.530167 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:17.518518    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca9efd32dda730d7cdef8f0334fdb0aa690b94d658b3fc607799f1a69ea14d32"
Oct 31 02:07:17.530167 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:17.518608    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"
Oct 31 02:07:22.246290 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.246257    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"]
Oct 31 02:07:22.246819 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.246805    5419 cpu_manager.go:414] "containerMap: removing container" podUID="02629c88-18e5-44a5-982d-a0240a62b90a" containerName="revoke-keys"
Oct 31 02:07:22.246866 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.246819    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02629c88-18e5-44a5-982d-a0240a62b90a" containerName="revoke-keys"
Oct 31 02:07:22.246917 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.246908    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="02629c88-18e5-44a5-982d-a0240a62b90a" containerName="revoke-keys"
Oct 31 02:07:22.250486 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.250471    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.303084 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.303022    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:22.303276 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.303266    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:07:22.343113 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.343082    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.343308 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.343260    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.343360 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.343334    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72n6\" (UniqueName: \"kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.352939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.352912    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:07:22.352939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.352926    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:07:22.355701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.355683    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"]
Oct 31 02:07:22.444057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.444027    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.444243 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.444081    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72n6\" (UniqueName: \"kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.444243 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.444126    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.444730 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.444704    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.446609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.446591    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.502508 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.502443    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72n6\" (UniqueName: \"kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.563251 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.563227    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:22.940265 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:22.940231    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"]
Oct 31 02:07:22.942225 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:22.942194    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa4f77e_412b_4d2b_b907_b0882658a0f6.slice/crio-db54506a87fa11400072a7ce14da63f8ded60e3edb26bfaf46a97718c0e77885 WatchSource:0}: Error finding container db54506a87fa11400072a7ce14da63f8ded60e3edb26bfaf46a97718c0e77885: Status 404 returned error can't find the container with id db54506a87fa11400072a7ce14da63f8ded60e3edb26bfaf46a97718c0e77885
Oct 31 02:07:23.557943 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:23.557852    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7" event={"ID":"4aa4f77e-412b-4d2b-b907-b0882658a0f6","Type":"ContainerStarted","Data":"5f4e7c42eb7b75a48aadb33f0542d4da57ad71b7db7f7aebdeff244c17111dd0"}
Oct 31 02:07:23.557943 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:23.557912    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7" event={"ID":"4aa4f77e-412b-4d2b-b907-b0882658a0f6","Type":"ContainerStarted","Data":"db54506a87fa11400072a7ce14da63f8ded60e3edb26bfaf46a97718c0e77885"}
Oct 31 02:07:23.674279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:23.674223    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7" podStartSLOduration=1.6742062469999999 podStartE2EDuration="1.674206247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:23.670693967 +0000 UTC m=+3764.971650544" watchObservedRunningTime="2026-10-31 02:07:23.674206247 +0000 UTC m=+3764.975162825"
Oct 31 02:07:24.584019 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.583986    5419 generic.go:356] "Generic (PLEG): container finished" podID="4aa4f77e-412b-4d2b-b907-b0882658a0f6" containerID="5f4e7c42eb7b75a48aadb33f0542d4da57ad71b7db7f7aebdeff244c17111dd0" exitCode=0
Oct 31 02:07:24.584472 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.584072    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7" event={"ID":"4aa4f77e-412b-4d2b-b907-b0882658a0f6","Type":"ContainerDied","Data":"5f4e7c42eb7b75a48aadb33f0542d4da57ad71b7db7f7aebdeff244c17111dd0"}
Oct 31 02:07:24.631489 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.631457    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"]
Oct 31 02:07:24.635441 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.635425    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.674061 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.674033    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqr4\" (UniqueName: \"kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.674210 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.674189    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.674275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.674217    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.690991 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.690961    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"]
Oct 31 02:07:24.775383 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.775352    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqr4\" (UniqueName: \"kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.775556 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.775437    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.775556 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.775459    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.776256 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.776225    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.778007 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.777959    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.832162 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.832114    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqr4\" (UniqueName: \"kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4\") pod \"maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:24.949113 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:24.949016    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:25.323085 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.323058    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"]
Oct 31 02:07:25.324929 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:25.324900    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea71697_c84e_4b6f_a129_c73ef1c79282.slice/crio-c29f4576a300f3b7accb8058fd701ca15bd3741815d16bc703346439a8b03ee2 WatchSource:0}: Error finding container c29f4576a300f3b7accb8058fd701ca15bd3741815d16bc703346439a8b03ee2: Status 404 returned error can't find the container with id c29f4576a300f3b7accb8058fd701ca15bd3741815d16bc703346439a8b03ee2
Oct 31 02:07:25.593126 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.593091    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj" event={"ID":"cea71697-c84e-4b6f-a129-c73ef1c79282","Type":"ContainerStarted","Data":"c29f4576a300f3b7accb8058fd701ca15bd3741815d16bc703346439a8b03ee2"}
Oct 31 02:07:25.717406 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.717375    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:25.786586 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.786556    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6\" (UniqueName: \"kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6\") pod \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") "
Oct 31 02:07:25.786782 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.786695    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca\") pod \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") "
Oct 31 02:07:25.786865 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.786815    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca\") pod \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\" (UID: \"4aa4f77e-412b-4d2b-b907-b0882658a0f6\") "
Oct 31 02:07:25.787206 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.787170    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca" pod "4aa4f77e-412b-4d2b-b907-b0882658a0f6" (UID: "4aa4f77e-412b-4d2b-b907-b0882658a0f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:25.788864 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.788838    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6" pod "4aa4f77e-412b-4d2b-b907-b0882658a0f6" (UID: "4aa4f77e-412b-4d2b-b907-b0882658a0f6"). InnerVolumeSpecName "kube-api-access-t72n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:25.789126 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.789108    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca" pod "4aa4f77e-412b-4d2b-b907-b0882658a0f6" (UID: "4aa4f77e-412b-4d2b-b907-b0882658a0f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:25.889177 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.889080    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:25.889177 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.889104    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72n6\" (UniqueName: \"kubernetes.io/projected/4aa4f77e-412b-4d2b-b907-b0882658a0f6-kube-api-access-t72n6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:25.889177 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:25.889115    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aa4f77e-412b-4d2b-b907-b0882658a0f6-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:26.602189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:26.602115    5419 generic.go:356] "Generic (PLEG): container finished" podID="cea71697-c84e-4b6f-a129-c73ef1c79282" containerID="e0269fbceb91de0a87711b3094b5cfe5405aa96292aa27ecca13c296e8a10dae" exitCode=0
Oct 31 02:07:26.602608 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:26.602198    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj" event={"ID":"cea71697-c84e-4b6f-a129-c73ef1c79282","Type":"ContainerDied","Data":"e0269fbceb91de0a87711b3094b5cfe5405aa96292aa27ecca13c296e8a10dae"}
Oct 31 02:07:26.603628 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:26.603603    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"
Oct 31 02:07:26.603753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:26.603641    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7" event={"ID":"4aa4f77e-412b-4d2b-b907-b0882658a0f6","Type":"ContainerDied","Data":"db54506a87fa11400072a7ce14da63f8ded60e3edb26bfaf46a97718c0e77885"}
Oct 31 02:07:26.603753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:26.603680    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db54506a87fa11400072a7ce14da63f8ded60e3edb26bfaf46a97718c0e77885"
Oct 31 02:07:27.558208 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.558115    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj" podUID="07283a5c-2baf-4c5a-90dc-754fe229218d" containerName="main" containerID="cri-o://116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6" gracePeriod=45
Oct 31 02:07:27.771177 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.771113    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:27.812594 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.812516    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca\") pod \"cea71697-c84e-4b6f-a129-c73ef1c79282\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") "
Oct 31 02:07:27.812734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.812603    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca\") pod \"cea71697-c84e-4b6f-a129-c73ef1c79282\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") "
Oct 31 02:07:27.812828 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.812808    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4\" (UniqueName: \"kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4\") pod \"cea71697-c84e-4b6f-a129-c73ef1c79282\" (UID: \"cea71697-c84e-4b6f-a129-c73ef1c79282\") "
Oct 31 02:07:27.812888 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.812851    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca" pod "cea71697-c84e-4b6f-a129-c73ef1c79282" (UID: "cea71697-c84e-4b6f-a129-c73ef1c79282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:27.813466 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.813366    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.815056 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.815035    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca" pod "cea71697-c84e-4b6f-a129-c73ef1c79282" (UID: "cea71697-c84e-4b6f-a129-c73ef1c79282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:27.815191 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.815173    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4" pod "cea71697-c84e-4b6f-a129-c73ef1c79282" (UID: "cea71697-c84e-4b6f-a129-c73ef1c79282"). InnerVolumeSpecName "kube-api-access-kdqr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:27.833799 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.833774    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:07:27.913966 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.913940    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm\") pod \"07283a5c-2baf-4c5a-90dc-754fe229218d\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") "
Oct 31 02:07:27.914176 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914042    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home\") pod \"07283a5c-2baf-4c5a-90dc-754fe229218d\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") "
Oct 31 02:07:27.914176 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914080    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir\") pod \"07283a5c-2baf-4c5a-90dc-754fe229218d\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") "
Oct 31 02:07:27.914314 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914174    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c\" (UniqueName: \"kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c\") pod \"07283a5c-2baf-4c5a-90dc-754fe229218d\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") "
Oct 31 02:07:27.914372 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914325    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache\") pod \"07283a5c-2baf-4c5a-90dc-754fe229218d\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") "
Oct 31 02:07:27.914432 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914381    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home" pod "07283a5c-2baf-4c5a-90dc-754fe229218d" (UID: "07283a5c-2baf-4c5a-90dc-754fe229218d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:27.914432 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914400    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir" pod "07283a5c-2baf-4c5a-90dc-754fe229218d" (UID: "07283a5c-2baf-4c5a-90dc-754fe229218d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:27.914432 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914391    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs\" (UniqueName: \"kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs\") pod \"07283a5c-2baf-4c5a-90dc-754fe229218d\" (UID: \"07283a5c-2baf-4c5a-90dc-754fe229218d\") "
Oct 31 02:07:27.914664 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914639    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache" pod "07283a5c-2baf-4c5a-90dc-754fe229218d" (UID: "07283a5c-2baf-4c5a-90dc-754fe229218d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:27.914988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914851    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cea71697-c84e-4b6f-a129-c73ef1c79282-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.914988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914880    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.914988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914895    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.914988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914909    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqr4\" (UniqueName: \"kubernetes.io/projected/cea71697-c84e-4b6f-a129-c73ef1c79282-kube-api-access-kdqr4\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.914988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.914925    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:27.917392 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.917371    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm" pod "07283a5c-2baf-4c5a-90dc-754fe229218d" (UID: "07283a5c-2baf-4c5a-90dc-754fe229218d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:27.917723 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.917706    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs" pod "07283a5c-2baf-4c5a-90dc-754fe229218d" (UID: "07283a5c-2baf-4c5a-90dc-754fe229218d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:27.918321 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:27.918302    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c" pod "07283a5c-2baf-4c5a-90dc-754fe229218d" (UID: "07283a5c-2baf-4c5a-90dc-754fe229218d"). InnerVolumeSpecName "kube-api-access-84l5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:28.016109 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.016080    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84l5c\" (UniqueName: \"kubernetes.io/projected/07283a5c-2baf-4c5a-90dc-754fe229218d-kube-api-access-84l5c\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:28.016109 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.016108    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07283a5c-2baf-4c5a-90dc-754fe229218d-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:28.016315 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.016119    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07283a5c-2baf-4c5a-90dc-754fe229218d-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:28.620131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.620096    5419 generic.go:356] "Generic (PLEG): container finished" podID="07283a5c-2baf-4c5a-90dc-754fe229218d" containerID="116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6" exitCode=0
Oct 31 02:07:28.620329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.620162    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"
Oct 31 02:07:28.620329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.620183    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj" event={"ID":"07283a5c-2baf-4c5a-90dc-754fe229218d","Type":"ContainerDied","Data":"116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6"}
Oct 31 02:07:28.620329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.620232    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj" event={"ID":"07283a5c-2baf-4c5a-90dc-754fe229218d","Type":"ContainerDied","Data":"7fc65d484e8276b65f1d2a01d47fd614d3329d5c3b419b6ddfeefdfb507a5b90"}
Oct 31 02:07:28.620329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.620250    5419 scope.go:122] "RemoveContainer" containerID="116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6"
Oct 31 02:07:28.621847 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.621820    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"
Oct 31 02:07:28.621847 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.621826    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj" event={"ID":"cea71697-c84e-4b6f-a129-c73ef1c79282","Type":"ContainerDied","Data":"c29f4576a300f3b7accb8058fd701ca15bd3741815d16bc703346439a8b03ee2"}
Oct 31 02:07:28.622005 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.621869    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29f4576a300f3b7accb8058fd701ca15bd3741815d16bc703346439a8b03ee2"
Oct 31 02:07:28.631716 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.631697    5419 scope.go:122] "RemoveContainer" containerID="116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6"
Oct 31 02:07:28.632089 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:28.631984    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6\": container with ID starting with 116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6 not found: ID does not exist" containerID="116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6"
Oct 31 02:07:28.632089 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.632014    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6"} err="failed to get container status \"116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6\": rpc error: code = NotFound desc = could not find container \"116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6\": container with ID starting with 116be5459731d723fd6b86c49b584a65d83aef84b3ccb1b9da5241a09fb797a6 not found: ID does not exist"
Oct 31 02:07:28.782310 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.782276    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"]
Oct 31 02:07:28.792233 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.787712    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c4d438/auth-test-model-c4d438-kserve-86f5f9577c-m5nbj"]
Oct 31 02:07:28.838920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:28.838845    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07283a5c-2baf-4c5a-90dc-754fe229218d" path="/var/lib/kubelet/pods/07283a5c-2baf-4c5a-90dc-754fe229218d/volumes"
Oct 31 02:07:29.060353 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:29.060317    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"]
Oct 31 02:07:29.069023 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:29.068981    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"]
Oct 31 02:07:29.080908 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:29.080872    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"]
Oct 31 02:07:33.735682 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:33.735645    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"]
Oct 31 02:07:33.842520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:33.842488    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"]
Oct 31 02:07:36.163736 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:36.163707    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"]
Oct 31 02:07:36.271072 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:36.271034    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"]
Oct 31 02:07:38.740829 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:38.737209    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" podUID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" containerName="payload-processing" containerID="cri-o://788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab" gracePeriod=25
Oct 31 02:07:38.844002 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:38.843961    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw" podUID="5f6558d2-2bfa-4ace-af46-60e0c3b41415" containerName="payload-pre-processing" containerID="cri-o://c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b" gracePeriod=25
Oct 31 02:07:38.844471 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:38.844447    5419 logging.go:55] [core] [Channel #13346 SubChannel #13348]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.13:50052", ServerName: "10.133.1.13:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.13:50052: connect: connection refused"
Oct 31 02:07:39.126243 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.126219    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 02:07:39.150806 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.150785    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 02:07:39.260087 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.260052    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume\") pod \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") "
Oct 31 02:07:39.260307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.260205    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume\") pod \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") "
Oct 31 02:07:39.260307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.260237    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n\" (UniqueName: \"kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n\") pod \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\" (UID: \"5f6558d2-2bfa-4ace-af46-60e0c3b41415\") "
Oct 31 02:07:39.260307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.260258    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8\" (UniqueName: \"kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8\") pod \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\" (UID: \"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99\") "
Oct 31 02:07:39.260513 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.260446    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume" pod "fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" (UID: "fa3858db-bc3a-41e1-ac81-4ef76d0cdb99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:39.260662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.260639    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume" pod "5f6558d2-2bfa-4ace-af46-60e0c3b41415" (UID: "5f6558d2-2bfa-4ace-af46-60e0c3b41415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:39.262642 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.262581    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8" pod "fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" (UID: "fa3858db-bc3a-41e1-ac81-4ef76d0cdb99"). InnerVolumeSpecName "kube-api-access-62gz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:39.262642 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.262595    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n" pod "5f6558d2-2bfa-4ace-af46-60e0c3b41415" (UID: "5f6558d2-2bfa-4ace-af46-60e0c3b41415"). InnerVolumeSpecName "kube-api-access-mhq2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:39.362275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.362241    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f6558d2-2bfa-4ace-af46-60e0c3b41415-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:39.362275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.362275    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhq2n\" (UniqueName: \"kubernetes.io/projected/5f6558d2-2bfa-4ace-af46-60e0c3b41415-kube-api-access-mhq2n\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:39.362467 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.362290    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62gz8\" (UniqueName: \"kubernetes.io/projected/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-kube-api-access-62gz8\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:39.362467 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.362303    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:39.705675 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.705589    5419 generic.go:356] "Generic (PLEG): container finished" podID="5f6558d2-2bfa-4ace-af46-60e0c3b41415" containerID="c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b" exitCode=0
Oct 31 02:07:39.705675 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.705634    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"
Oct 31 02:07:39.705948 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.705680    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw" event={"ID":"5f6558d2-2bfa-4ace-af46-60e0c3b41415","Type":"ContainerDied","Data":"c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b"}
Oct 31 02:07:39.705948 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.705737    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw" event={"ID":"5f6558d2-2bfa-4ace-af46-60e0c3b41415","Type":"ContainerDied","Data":"f8e0f64348b2f0f7258fc78e1a688b3fa9d396f2a5f076603facde67f497bb69"}
Oct 31 02:07:39.705948 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.705766    5419 scope.go:122] "RemoveContainer" containerID="c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b"
Oct 31 02:07:39.707261 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.707236    5419 generic.go:356] "Generic (PLEG): container finished" podID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" containerID="788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab" exitCode=0
Oct 31 02:07:39.707402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.707283    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" event={"ID":"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99","Type":"ContainerDied","Data":"788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab"}
Oct 31 02:07:39.707402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.707326    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" event={"ID":"fa3858db-bc3a-41e1-ac81-4ef76d0cdb99","Type":"ContainerDied","Data":"d7f44ee69281692098487bb166fa6fd8d38678fa846cae9d7367fa6abab2d418"}
Oct 31 02:07:39.707402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.707324    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"
Oct 31 02:07:39.718752 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.718658    5419 scope.go:122] "RemoveContainer" containerID="c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b"
Oct 31 02:07:39.719011 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:39.718986    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b\": container with ID starting with c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b not found: ID does not exist" containerID="c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b"
Oct 31 02:07:39.719095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.719021    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b"} err="failed to get container status \"c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b\": rpc error: code = NotFound desc = could not find container \"c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b\": container with ID starting with c722fc2a0b560f35d63a30eef448a870d94f2c9b0ba4d87b6bd1acf41739782b not found: ID does not exist"
Oct 31 02:07:39.719095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.719044    5419 scope.go:122] "RemoveContainer" containerID="788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab"
Oct 31 02:07:39.728266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.728248    5419 scope.go:122] "RemoveContainer" containerID="788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab"
Oct 31 02:07:39.728529 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:39.728507    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab\": container with ID starting with 788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab not found: ID does not exist" containerID="788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab"
Oct 31 02:07:39.728574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.728537    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab"} err="failed to get container status \"788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab\": rpc error: code = NotFound desc = could not find container \"788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab\": container with ID starting with 788c7e2c54f30e9a5dec60ff8829d631373af736d415ecd68dc037324c54feab not found: ID does not exist"
Oct 31 02:07:39.844911 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.844861    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp" podUID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.13:50052\" within 1s: context deadline exceeded"
Oct 31 02:07:39.868511 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.868478    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"]
Oct 31 02:07:39.917997 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.917957    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-98c748-65f9f5d899-7hrbw"]
Oct 31 02:07:39.980130 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.980097    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"]
Oct 31 02:07:39.985751 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:39.985725    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-98c748-c9855bdb9-k4vmp"]
Oct 31 02:07:40.045520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.045490    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"]
Oct 31 02:07:40.048069 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.048048    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b658cd-bwkp8"]
Oct 31 02:07:40.627419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.627386    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"]
Oct 31 02:07:40.676427 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.676401    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-051514-5bz6w"]
Oct 31 02:07:40.837171 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.837123    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02629c88-18e5-44a5-982d-a0240a62b90a" path="/var/lib/kubelet/pods/02629c88-18e5-44a5-982d-a0240a62b90a/volumes"
Oct 31 02:07:40.837536 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.837523    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6558d2-2bfa-4ace-af46-60e0c3b41415" path="/var/lib/kubelet/pods/5f6558d2-2bfa-4ace-af46-60e0c3b41415/volumes"
Oct 31 02:07:40.837889 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.837878    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d750ebd-3909-431d-8031-a42a152bcf2b" path="/var/lib/kubelet/pods/6d750ebd-3909-431d-8031-a42a152bcf2b/volumes"
Oct 31 02:07:40.838394 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:40.838372    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" path="/var/lib/kubelet/pods/fa3858db-bc3a-41e1-ac81-4ef76d0cdb99/volumes"
Oct 31 02:07:41.164832 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.164795    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerName="payload-processing" containerID="cri-o://10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e" gracePeriod=25
Oct 31 02:07:41.213779 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.213746    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"]
Oct 31 02:07:41.214442 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214420    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5f6558d2-2bfa-4ace-af46-60e0c3b41415" containerName="payload-pre-processing"
Oct 31 02:07:41.214442 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214439    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6558d2-2bfa-4ace-af46-60e0c3b41415" containerName="payload-pre-processing"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214455    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4aa4f77e-412b-4d2b-b907-b0882658a0f6" containerName="revoke-keys"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214463    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa4f77e-412b-4d2b-b907-b0882658a0f6" containerName="revoke-keys"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214476    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cea71697-c84e-4b6f-a129-c73ef1c79282" containerName="revoke-keys"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214485    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea71697-c84e-4b6f-a129-c73ef1c79282" containerName="revoke-keys"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214499    5419 cpu_manager.go:414] "containerMap: removing container" podUID="07283a5c-2baf-4c5a-90dc-754fe229218d" containerName="main"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214506    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07283a5c-2baf-4c5a-90dc-754fe229218d" containerName="main"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214519    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" containerName="payload-processing"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214524    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" containerName="payload-processing"
Oct 31 02:07:41.214649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214646    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6558d2-2bfa-4ace-af46-60e0c3b41415" containerName="payload-pre-processing"
Oct 31 02:07:41.215094 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214657    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa4f77e-412b-4d2b-b907-b0882658a0f6" containerName="revoke-keys"
Oct 31 02:07:41.215094 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214664    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea71697-c84e-4b6f-a129-c73ef1c79282" containerName="revoke-keys"
Oct 31 02:07:41.215094 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214669    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="07283a5c-2baf-4c5a-90dc-754fe229218d" containerName="main"
Oct 31 02:07:41.215094 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.214676    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3858db-bc3a-41e1-ac81-4ef76d0cdb99" containerName="payload-processing"
Oct 31 02:07:41.219598 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.219571    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.271925 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.271882    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerName="payload-pre-processing" containerID="cri-o://26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15" gracePeriod=25
Oct 31 02:07:41.273774 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.273749    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"]
Oct 31 02:07:41.293384 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.293361    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6nm\" (UniqueName: \"kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.293523 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.293399    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.293523 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.293426    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.319859 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.319834    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:07:41.319859 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.319834    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:41.320023 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.319884    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:07:41.320023 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.319901    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:07:41.395080 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.395044    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6nm\" (UniqueName: \"kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.395259 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.395091    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.395332 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.395286    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.395831 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.395802    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.397706 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.397686    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.452341 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.452269    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6nm\" (UniqueName: \"kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm\") pod \"maas-api-revoke-sub-b484d7ec-r8zkt\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.535043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.535012    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:41.937160 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:41.936411    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"]
Oct 31 02:07:42.730291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:42.730244    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt" event={"ID":"aba786b8-404f-4add-8a3c-b7ca53bca85d","Type":"ContainerStarted","Data":"33a809ac93061e4f63156a6133b98169c806786a16bea012c86dd54e5c962f40"}
Oct 31 02:07:42.730291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:42.730297    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt" event={"ID":"aba786b8-404f-4add-8a3c-b7ca53bca85d","Type":"ContainerStarted","Data":"24184deaedcb299e9b7b315bbf6d1c468a13005d12bb608c5f34fc4c574745ac"}
Oct 31 02:07:42.846603 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:42.846551    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt" podStartSLOduration=1.846537089 podStartE2EDuration="1.846537089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:42.844274873 +0000 UTC m=+3784.145231450" watchObservedRunningTime="2026-10-31 02:07:42.846537089 +0000 UTC m=+3784.147493665"
Oct 31 02:07:43.737853 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:43.737817    5419 generic.go:356] "Generic (PLEG): container finished" podID="aba786b8-404f-4add-8a3c-b7ca53bca85d" containerID="33a809ac93061e4f63156a6133b98169c806786a16bea012c86dd54e5c962f40" exitCode=0
Oct 31 02:07:43.738297 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:43.737886    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt" event={"ID":"aba786b8-404f-4add-8a3c-b7ca53bca85d","Type":"ContainerDied","Data":"33a809ac93061e4f63156a6133b98169c806786a16bea012c86dd54e5c962f40"}
Oct 31 02:07:44.083191 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.083056    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main" containerID="cri-o://1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0" gracePeriod=45
Oct 31 02:07:44.089806 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.089766    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" containerID="cri-o://c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81" gracePeriod=45
Oct 31 02:07:44.100757 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.100721    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main" containerID="cri-o://3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f" gracePeriod=45
Oct 31 02:07:44.150092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.150057    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused"
Oct 31 02:07:44.387282 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.387251    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 02:07:44.390667 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.390644    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 02:07:44.394323 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.394304    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 02:07:44.550405 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550376    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home\") pod \"dc80e2b1-5791-4e07-acde-514653cd7385\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") "
Oct 31 02:07:44.550559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550413    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs\") pod \"ef5ecb7d-467c-469a-8c66-d79d6149b570\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") "
Oct 31 02:07:44.550559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550478    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs\") pod \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") "
Oct 31 02:07:44.550559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550526    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache\") pod \"ef5ecb7d-467c-469a-8c66-d79d6149b570\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") "
Oct 31 02:07:44.550750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550571    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache\") pod \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") "
Oct 31 02:07:44.550750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550611    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir\") pod \"dc80e2b1-5791-4e07-acde-514653cd7385\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") "
Oct 31 02:07:44.550750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550644    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home\") pod \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") "
Oct 31 02:07:44.550750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550663    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home" pod "dc80e2b1-5791-4e07-acde-514653cd7385" (UID: "dc80e2b1-5791-4e07-acde-514653cd7385"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.550750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550700    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb\" (UniqueName: \"kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb\") pod \"dc80e2b1-5791-4e07-acde-514653cd7385\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") "
Oct 31 02:07:44.550750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550733    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm\") pod \"ef5ecb7d-467c-469a-8c66-d79d6149b570\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") "
Oct 31 02:07:44.551080 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550785    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c\" (UniqueName: \"kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c\") pod \"ef5ecb7d-467c-469a-8c66-d79d6149b570\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") "
Oct 31 02:07:44.551080 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550833    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home\") pod \"ef5ecb7d-467c-469a-8c66-d79d6149b570\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") "
Oct 31 02:07:44.551080 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550853    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache" pod "3041fe71-9784-4551-a2a4-4d48a4a10ae0" (UID: "3041fe71-9784-4551-a2a4-4d48a4a10ae0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.551080 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.550864    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache" pod "ef5ecb7d-467c-469a-8c66-d79d6149b570" (UID: "ef5ecb7d-467c-469a-8c66-d79d6149b570"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.551303 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551098    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir" pod "dc80e2b1-5791-4e07-acde-514653cd7385" (UID: "dc80e2b1-5791-4e07-acde-514653cd7385"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.551350 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551314    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home" pod "3041fe71-9784-4551-a2a4-4d48a4a10ae0" (UID: "3041fe71-9784-4551-a2a4-4d48a4a10ae0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.551350 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551316    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home" pod "ef5ecb7d-467c-469a-8c66-d79d6149b570" (UID: "ef5ecb7d-467c-469a-8c66-d79d6149b570"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.551461 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551389    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir\") pod \"ef5ecb7d-467c-469a-8c66-d79d6149b570\" (UID: \"ef5ecb7d-467c-469a-8c66-d79d6149b570\") "
Oct 31 02:07:44.551461 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551440    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm\") pod \"dc80e2b1-5791-4e07-acde-514653cd7385\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") "
Oct 31 02:07:44.551566 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551477    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm\") pod \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") "
Oct 31 02:07:44.551566 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551529    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6\" (UniqueName: \"kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6\") pod \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") "
Oct 31 02:07:44.551686 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551588    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir\") pod \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\" (UID: \"3041fe71-9784-4551-a2a4-4d48a4a10ae0\") "
Oct 31 02:07:44.551686 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551648    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs\") pod \"dc80e2b1-5791-4e07-acde-514653cd7385\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") "
Oct 31 02:07:44.551781 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551681    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache\") pod \"dc80e2b1-5791-4e07-acde-514653cd7385\" (UID: \"dc80e2b1-5791-4e07-acde-514653cd7385\") "
Oct 31 02:07:44.551781 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.551701    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir" pod "ef5ecb7d-467c-469a-8c66-d79d6149b570" (UID: "ef5ecb7d-467c-469a-8c66-d79d6149b570"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.552242 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552217    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir" pod "3041fe71-9784-4551-a2a4-4d48a4a10ae0" (UID: "3041fe71-9784-4551-a2a4-4d48a4a10ae0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552480    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552510    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552525    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552539    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552551    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552563    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552576    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.552722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.552590    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.554194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.554123    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs" pod "ef5ecb7d-467c-469a-8c66-d79d6149b570" (UID: "ef5ecb7d-467c-469a-8c66-d79d6149b570"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:44.554194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.554176    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache" pod "dc80e2b1-5791-4e07-acde-514653cd7385" (UID: "dc80e2b1-5791-4e07-acde-514653cd7385"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.554528 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.554447    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm" pod "ef5ecb7d-467c-469a-8c66-d79d6149b570" (UID: "ef5ecb7d-467c-469a-8c66-d79d6149b570"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.555042 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.555022    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6" pod "3041fe71-9784-4551-a2a4-4d48a4a10ae0" (UID: "3041fe71-9784-4551-a2a4-4d48a4a10ae0"). InnerVolumeSpecName "kube-api-access-zddx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:44.556333 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.556308    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm" pod "dc80e2b1-5791-4e07-acde-514653cd7385" (UID: "dc80e2b1-5791-4e07-acde-514653cd7385"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.556526 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.556500    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs" pod "3041fe71-9784-4551-a2a4-4d48a4a10ae0" (UID: "3041fe71-9784-4551-a2a4-4d48a4a10ae0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:44.556526 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.556511    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb" pod "dc80e2b1-5791-4e07-acde-514653cd7385" (UID: "dc80e2b1-5791-4e07-acde-514653cd7385"). InnerVolumeSpecName "kube-api-access-ktlzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:44.557057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.557023    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm" pod "3041fe71-9784-4551-a2a4-4d48a4a10ae0" (UID: "3041fe71-9784-4551-a2a4-4d48a4a10ae0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:44.557310 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.557287    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs" pod "dc80e2b1-5791-4e07-acde-514653cd7385" (UID: "dc80e2b1-5791-4e07-acde-514653cd7385"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:44.558286 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.558259    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c" pod "ef5ecb7d-467c-469a-8c66-d79d6149b570" (UID: "ef5ecb7d-467c-469a-8c66-d79d6149b570"). InnerVolumeSpecName "kube-api-access-snf2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:44.591829 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:44.591793    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654156    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktlzb\" (UniqueName: \"kubernetes.io/projected/dc80e2b1-5791-4e07-acde-514653cd7385-kube-api-access-ktlzb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654183    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef5ecb7d-467c-469a-8c66-d79d6149b570-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654194    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snf2c\" (UniqueName: \"kubernetes.io/projected/ef5ecb7d-467c-469a-8c66-d79d6149b570-kube-api-access-snf2c\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654202    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654211    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3041fe71-9784-4551-a2a4-4d48a4a10ae0-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654219    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddx6\" (UniqueName: \"kubernetes.io/projected/3041fe71-9784-4551-a2a4-4d48a4a10ae0-kube-api-access-zddx6\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654227    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc80e2b1-5791-4e07-acde-514653cd7385-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654239    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc80e2b1-5791-4e07-acde-514653cd7385-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654247    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef5ecb7d-467c-469a-8c66-d79d6149b570-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.654847 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.654255    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3041fe71-9784-4551-a2a4-4d48a4a10ae0-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:44.747866 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.747834    5419 generic.go:356] "Generic (PLEG): container finished" podID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerID="c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81" exitCode=0
Oct 31 02:07:44.748366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.747897    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"
Oct 31 02:07:44.748366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.747908    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" event={"ID":"3041fe71-9784-4551-a2a4-4d48a4a10ae0","Type":"ContainerDied","Data":"c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81"}
Oct 31 02:07:44.748366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.747950    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj" event={"ID":"3041fe71-9784-4551-a2a4-4d48a4a10ae0","Type":"ContainerDied","Data":"32304d958f6ace6a96dbb5904a804a6a4a0159580027e1d78fbb62ade243683f"}
Oct 31 02:07:44.748366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.747971    5419 scope.go:122] "RemoveContainer" containerID="c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81"
Oct 31 02:07:44.757219 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.757172    5419 generic.go:356] "Generic (PLEG): container finished" podID="dc80e2b1-5791-4e07-acde-514653cd7385" containerID="1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0" exitCode=0
Oct 31 02:07:44.757372 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.757279    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"
Oct 31 02:07:44.757372 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.757295    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" event={"ID":"dc80e2b1-5791-4e07-acde-514653cd7385","Type":"ContainerDied","Data":"1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0"}
Oct 31 02:07:44.757372 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.757328    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5" event={"ID":"dc80e2b1-5791-4e07-acde-514653cd7385","Type":"ContainerDied","Data":"9887c12fc21666d17b6b1967c668cd55535655ae02d185fa01761b62abf43d6d"}
Oct 31 02:07:44.760030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.759463    5419 generic.go:356] "Generic (PLEG): container finished" podID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerID="3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f" exitCode=0
Oct 31 02:07:44.760030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.759573    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"
Oct 31 02:07:44.760030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.759659    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" event={"ID":"ef5ecb7d-467c-469a-8c66-d79d6149b570","Type":"ContainerDied","Data":"3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f"}
Oct 31 02:07:44.760030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.759692    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj" event={"ID":"ef5ecb7d-467c-469a-8c66-d79d6149b570","Type":"ContainerDied","Data":"cd7d7390234359b93142c2c45f599665888f45ba65cb4fdf527e4ee195a074c9"}
Oct 31 02:07:44.762129 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.762109    5419 scope.go:122] "RemoveContainer" containerID="c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81"
Oct 31 02:07:44.762559 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:44.762534    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81\": container with ID starting with c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81 not found: ID does not exist" containerID="c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81"
Oct 31 02:07:44.762629 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.762571    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81"} err="failed to get container status \"c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81\": rpc error: code = NotFound desc = could not find container \"c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81\": container with ID starting with c96277cb9cce0f262b7528e67d1820458c1f9d7fb231ea6d2fcc81eae7d43a81 not found: ID does not exist"
Oct 31 02:07:44.762629 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.762593    5419 scope.go:122] "RemoveContainer" containerID="1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0"
Oct 31 02:07:44.774306 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.774284    5419 scope.go:122] "RemoveContainer" containerID="1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0"
Oct 31 02:07:44.774692 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:44.774666    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0\": container with ID starting with 1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0 not found: ID does not exist" containerID="1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0"
Oct 31 02:07:44.774783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.774704    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0"} err="failed to get container status \"1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0\": rpc error: code = NotFound desc = could not find container \"1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0\": container with ID starting with 1c84a761ce50780ea8b6d48f4cd46be0602708329d27553e8abf9226d21541b0 not found: ID does not exist"
Oct 31 02:07:44.774783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.774728    5419 scope.go:122] "RemoveContainer" containerID="3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f"
Oct 31 02:07:44.788196 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.788170    5419 scope.go:122] "RemoveContainer" containerID="3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f"
Oct 31 02:07:44.788552 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:44.788522    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f\": container with ID starting with 3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f not found: ID does not exist" containerID="3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f"
Oct 31 02:07:44.788662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.788563    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f"} err="failed to get container status \"3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f\": rpc error: code = NotFound desc = could not find container \"3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f\": container with ID starting with 3e3bed362bc3269ed762b60598263b08bc991f11f3392023f8a571ba5c68ff2f not found: ID does not exist"
Oct 31 02:07:44.892079 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.892054    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:44.910790 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.910721    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"]
Oct 31 02:07:44.913153 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:44.913113    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/simulator-w4-8f6b5b-kserve-f447d6c7b-p6tvj"]
Oct 31 02:07:45.021839 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.021801    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"]
Oct 31 02:07:45.029928 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.029859    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"]
Oct 31 02:07:45.030881 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.030852    5419 cpu_manager.go:414] "containerMap: removing container" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main"
Oct 31 02:07:45.030881 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.030877    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main"
Oct 31 02:07:45.031097 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.030893    5419 cpu_manager.go:414] "containerMap: removing container" podUID="aba786b8-404f-4add-8a3c-b7ca53bca85d" containerName="revoke-keys"
Oct 31 02:07:45.031097 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.030901    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba786b8-404f-4add-8a3c-b7ca53bca85d" containerName="revoke-keys"
Oct 31 02:07:45.031097 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.030919    5419 cpu_manager.go:414] "containerMap: removing container" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main"
Oct 31 02:07:45.031097 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.030926    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main"
Oct 31 02:07:45.031097 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.031003    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main"
Oct 31 02:07:45.031097 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.031010    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main"
Oct 31 02:07:45.031565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.031328    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" containerName="main"
Oct 31 02:07:45.031565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.031365    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" containerName="main"
Oct 31 02:07:45.031565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.031384    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba786b8-404f-4add-8a3c-b7ca53bca85d" containerName="revoke-keys"
Oct 31 02:07:45.031565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.031394    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" containerName="main"
Oct 31 02:07:45.035346 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.035324    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/distinct-w4-8f6b5b-kserve-66c6c44cb-hrfh5"]
Oct 31 02:07:45.035577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.035562    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.060971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.060949    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca\") pod \"aba786b8-404f-4add-8a3c-b7ca53bca85d\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") "
Oct 31 02:07:45.061116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.061102    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm\" (UniqueName: \"kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm\") pod \"aba786b8-404f-4add-8a3c-b7ca53bca85d\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") "
Oct 31 02:07:45.061259 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.061244    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca\") pod \"aba786b8-404f-4add-8a3c-b7ca53bca85d\" (UID: \"aba786b8-404f-4add-8a3c-b7ca53bca85d\") "
Oct 31 02:07:45.061373 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.061294    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca" pod "aba786b8-404f-4add-8a3c-b7ca53bca85d" (UID: "aba786b8-404f-4add-8a3c-b7ca53bca85d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:45.061600 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.061581    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:45.063469 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.063444    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm" pod "aba786b8-404f-4add-8a3c-b7ca53bca85d" (UID: "aba786b8-404f-4add-8a3c-b7ca53bca85d"). InnerVolumeSpecName "kube-api-access-ll6nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:45.063469 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.063461    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca" pod "aba786b8-404f-4add-8a3c-b7ca53bca85d" (UID: "aba786b8-404f-4add-8a3c-b7ca53bca85d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:45.074164 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.074127    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"]
Oct 31 02:07:45.078204 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.078188    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.084421 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.084402    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"]
Oct 31 02:07:45.134649 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.134619    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"]
Oct 31 02:07:45.141612 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.141580    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"]
Oct 31 02:07:45.144635 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.144611    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8f6b5b/premium-w4-8f6b5b-kserve-648dc97bb5-trpxj"]
Oct 31 02:07:45.162980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.162922    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.162980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.162954    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v5kx\" (UniqueName: \"kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.163109 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.163049    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bz44\" (UniqueName: \"kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.163173 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.163109    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.163226 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.163213    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.163298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.163286    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.163428 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.163410    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6nm\" (UniqueName: \"kubernetes.io/projected/aba786b8-404f-4add-8a3c-b7ca53bca85d-kube-api-access-ll6nm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:45.163466 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.163433    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba786b8-404f-4add-8a3c-b7ca53bca85d-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:45.264490 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.264453    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.264690 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.264531    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.264690 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.264558    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v5kx\" (UniqueName: \"kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.264690 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.264613    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bz44\" (UniqueName: \"kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.264690 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.264655    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.264991 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.264719    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.265293 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.265265    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.265497 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.265475    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.267757 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.267727    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.267869 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.267754    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.320890 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.320860    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bz44\" (UniqueName: \"kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44\") pod \"maas-api-revoke-sub-6c94b8bd-lljwp\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.321482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.321460    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v5kx\" (UniqueName: \"kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx\") pod \"maas-api-revoke-sub-063e7c83-bt5dx\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.349225 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.349202    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:45.394845 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.394817    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:45.723566 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.723541    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"]
Oct 31 02:07:45.725226 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:45.725189    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2a64d6_4066_4f81_82f9_50046d991664.slice/crio-32a7bea2fb8502021b9b589e03a5daea18ac0d627b77a686543de665e6567ae8 WatchSource:0}: Error finding container 32a7bea2fb8502021b9b589e03a5daea18ac0d627b77a686543de665e6567ae8: Status 404 returned error can't find the container with id 32a7bea2fb8502021b9b589e03a5daea18ac0d627b77a686543de665e6567ae8
Oct 31 02:07:45.766574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.766549    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp" event={"ID":"5d2a64d6-4066-4f81-82f9-50046d991664","Type":"ContainerStarted","Data":"32a7bea2fb8502021b9b589e03a5daea18ac0d627b77a686543de665e6567ae8"}
Oct 31 02:07:45.769767 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.769736    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt" event={"ID":"aba786b8-404f-4add-8a3c-b7ca53bca85d","Type":"ContainerDied","Data":"24184deaedcb299e9b7b315bbf6d1c468a13005d12bb608c5f34fc4c574745ac"}
Oct 31 02:07:45.769767 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.769764    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"
Oct 31 02:07:45.769926 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.769785    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24184deaedcb299e9b7b315bbf6d1c468a13005d12bb608c5f34fc4c574745ac"
Oct 31 02:07:45.785329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:45.785310    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"]
Oct 31 02:07:45.876328 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:45.876291    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ee14a8_4fc0_4696_a2e3_554a9b9f8086.slice/crio-6e079185bbe4988726f234cf00031f73c0b61afa4f7d00bae244ec8bcc2ebffd WatchSource:0}: Error finding container 6e079185bbe4988726f234cf00031f73c0b61afa4f7d00bae244ec8bcc2ebffd: Status 404 returned error can't find the container with id 6e079185bbe4988726f234cf00031f73c0b61afa4f7d00bae244ec8bcc2ebffd
Oct 31 02:07:46.779969 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.779931    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx" event={"ID":"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086","Type":"ContainerStarted","Data":"b922618ac23cfc8b5c0c4fd6ef074a1f8052cfc7dbdfc76bf0cb8cb21d99bfb4"}
Oct 31 02:07:46.780434 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.779993    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx" event={"ID":"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086","Type":"ContainerStarted","Data":"6e079185bbe4988726f234cf00031f73c0b61afa4f7d00bae244ec8bcc2ebffd"}
Oct 31 02:07:46.781850 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.781826    5419 generic.go:356] "Generic (PLEG): container finished" podID="5d2a64d6-4066-4f81-82f9-50046d991664" containerID="342fb5b7fe2bc37414c16457228325718811443ecadd7286d1e1dd4adb4a1ef8" exitCode=0
Oct 31 02:07:46.781978 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.781903    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp" event={"ID":"5d2a64d6-4066-4f81-82f9-50046d991664","Type":"ContainerDied","Data":"342fb5b7fe2bc37414c16457228325718811443ecadd7286d1e1dd4adb4a1ef8"}
Oct 31 02:07:46.838244 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.838209    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3041fe71-9784-4551-a2a4-4d48a4a10ae0" path="/var/lib/kubelet/pods/3041fe71-9784-4551-a2a4-4d48a4a10ae0/volumes"
Oct 31 02:07:46.838758 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.838740    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc80e2b1-5791-4e07-acde-514653cd7385" path="/var/lib/kubelet/pods/dc80e2b1-5791-4e07-acde-514653cd7385/volumes"
Oct 31 02:07:46.839115 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.839103    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5ecb7d-467c-469a-8c66-d79d6149b570" path="/var/lib/kubelet/pods/ef5ecb7d-467c-469a-8c66-d79d6149b570/volumes"
Oct 31 02:07:46.899418 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:46.899287    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx" podStartSLOduration=1.8992661709999998 podStartE2EDuration="1.899266171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:46.89710651 +0000 UTC m=+3788.198063123" watchObservedRunningTime="2026-10-31 02:07:46.899266171 +0000 UTC m=+3788.200222750"
Oct 31 02:07:47.172355 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.172277    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"]
Oct 31 02:07:47.172826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.172534    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy" containerID="cri-o://81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2" gracePeriod=30
Oct 31 02:07:47.411754 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.411711    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:47.411935 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.411711    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:47.790568 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.790533    5419 generic.go:356] "Generic (PLEG): container finished" podID="e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" containerID="b922618ac23cfc8b5c0c4fd6ef074a1f8052cfc7dbdfc76bf0cb8cb21d99bfb4" exitCode=0
Oct 31 02:07:47.790961 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.790612    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx" event={"ID":"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086","Type":"ContainerDied","Data":"b922618ac23cfc8b5c0c4fd6ef074a1f8052cfc7dbdfc76bf0cb8cb21d99bfb4"}
Oct 31 02:07:47.934817 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.934789    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:47.936252 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.936232    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca\") pod \"5d2a64d6-4066-4f81-82f9-50046d991664\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") "
Oct 31 02:07:47.936390 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.936376    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca\") pod \"5d2a64d6-4066-4f81-82f9-50046d991664\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") "
Oct 31 02:07:47.936472 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.936457    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44\" (UniqueName: \"kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44\") pod \"5d2a64d6-4066-4f81-82f9-50046d991664\" (UID: \"5d2a64d6-4066-4f81-82f9-50046d991664\") "
Oct 31 02:07:47.936769 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.936742    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca" pod "5d2a64d6-4066-4f81-82f9-50046d991664" (UID: "5d2a64d6-4066-4f81-82f9-50046d991664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:47.937106 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.937085    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:47.938528 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.938505    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca" pod "5d2a64d6-4066-4f81-82f9-50046d991664" (UID: "5d2a64d6-4066-4f81-82f9-50046d991664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:47.938734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:47.938718    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44" pod "5d2a64d6-4066-4f81-82f9-50046d991664" (UID: "5d2a64d6-4066-4f81-82f9-50046d991664"). InnerVolumeSpecName "kube-api-access-2bz44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:48.038679 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.038645    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bz44\" (UniqueName: \"kubernetes.io/projected/5d2a64d6-4066-4f81-82f9-50046d991664-kube-api-access-2bz44\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:48.038679 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.038673    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2a64d6-4066-4f81-82f9-50046d991664-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:48.045122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.045053    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"]
Oct 31 02:07:48.045630 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.045616    5419 cpu_manager.go:414] "containerMap: removing container" podUID="5d2a64d6-4066-4f81-82f9-50046d991664" containerName="revoke-keys"
Oct 31 02:07:48.045676 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.045632    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2a64d6-4066-4f81-82f9-50046d991664" containerName="revoke-keys"
Oct 31 02:07:48.045738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.045729    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2a64d6-4066-4f81-82f9-50046d991664" containerName="revoke-keys"
Oct 31 02:07:48.049229 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.049205    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.105633 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.105555    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"]
Oct 31 02:07:48.139815 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.139786    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pd4p\" (UniqueName: \"kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.139815 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.139828    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.140017 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.139991    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.240706 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.240671    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.240902 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.240738    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pd4p\" (UniqueName: \"kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.240902 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.240773    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.241535 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.241510    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.243216 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.243197    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.297443 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.297382    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pd4p\" (UniqueName: \"kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.366360 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.366334    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:48.739787 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.739754    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"]
Oct 31 02:07:48.741177 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:48.741126    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab42a280_8ba3_4a41_bec2_533472dea182.slice/crio-c1c8e6ed2d1241c5e31e1e1fb566acdd116056a9c86d1cf16c51d0a4ebb18108 WatchSource:0}: Error finding container c1c8e6ed2d1241c5e31e1e1fb566acdd116056a9c86d1cf16c51d0a4ebb18108: Status 404 returned error can't find the container with id c1c8e6ed2d1241c5e31e1e1fb566acdd116056a9c86d1cf16c51d0a4ebb18108
Oct 31 02:07:48.797744 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.797711    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n" event={"ID":"ab42a280-8ba3-4a41-bec2-533472dea182","Type":"ContainerStarted","Data":"c1c8e6ed2d1241c5e31e1e1fb566acdd116056a9c86d1cf16c51d0a4ebb18108"}
Oct 31 02:07:48.799191 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.799169    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"
Oct 31 02:07:48.799289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.799173    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp" event={"ID":"5d2a64d6-4066-4f81-82f9-50046d991664","Type":"ContainerDied","Data":"32a7bea2fb8502021b9b589e03a5daea18ac0d627b77a686543de665e6567ae8"}
Oct 31 02:07:48.799289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.799247    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32a7bea2fb8502021b9b589e03a5daea18ac0d627b77a686543de665e6567ae8"
Oct 31 02:07:48.941564 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.941527    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:48.947731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.947709    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca\") pod \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") "
Oct 31 02:07:48.947897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.947881    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca\") pod \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") "
Oct 31 02:07:48.947980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.947967    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx\" (UniqueName: \"kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx\") pod \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\" (UID: \"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086\") "
Oct 31 02:07:48.948233 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.948206    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca" pod "e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" (UID: "e8ee14a8-4fc0-4696-a2e3-554a9b9f8086"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:48.948514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.948322    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:48.949837 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.949815    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca" pod "e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" (UID: "e8ee14a8-4fc0-4696-a2e3-554a9b9f8086"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:48.950306 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:48.950291    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx" pod "e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" (UID: "e8ee14a8-4fc0-4696-a2e3-554a9b9f8086"). InnerVolumeSpecName "kube-api-access-2v5kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:49.048981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.048906    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:49.048981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.048935    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v5kx\" (UniqueName: \"kubernetes.io/projected/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086-kube-api-access-2v5kx\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:49.807342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.807249    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"
Oct 31 02:07:49.807342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.807265    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx" event={"ID":"e8ee14a8-4fc0-4696-a2e3-554a9b9f8086","Type":"ContainerDied","Data":"6e079185bbe4988726f234cf00031f73c0b61afa4f7d00bae244ec8bcc2ebffd"}
Oct 31 02:07:49.807342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.807319    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e079185bbe4988726f234cf00031f73c0b61afa4f7d00bae244ec8bcc2ebffd"
Oct 31 02:07:49.809151 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.809111    5419 generic.go:356] "Generic (PLEG): container finished" podID="ab42a280-8ba3-4a41-bec2-533472dea182" containerID="f72e8558cfcfa239cd9db1728148b8b103adfcf170ec1caef5be94647b132ff8" exitCode=0
Oct 31 02:07:49.809276 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:49.809194    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n" event={"ID":"ab42a280-8ba3-4a41-bec2-533472dea182","Type":"ContainerDied","Data":"f72e8558cfcfa239cd9db1728148b8b103adfcf170ec1caef5be94647b132ff8"}
Oct 31 02:07:50.957816 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.957791    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:50.967021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.966999    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca\") pod \"ab42a280-8ba3-4a41-bec2-533472dea182\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") "
Oct 31 02:07:50.967121 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.967036    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p\" (UniqueName: \"kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p\") pod \"ab42a280-8ba3-4a41-bec2-533472dea182\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") "
Oct 31 02:07:50.967121 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.967070    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca\") pod \"ab42a280-8ba3-4a41-bec2-533472dea182\" (UID: \"ab42a280-8ba3-4a41-bec2-533472dea182\") "
Oct 31 02:07:50.967430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.967408    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca" pod "ab42a280-8ba3-4a41-bec2-533472dea182" (UID: "ab42a280-8ba3-4a41-bec2-533472dea182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:50.969189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.969167    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca" pod "ab42a280-8ba3-4a41-bec2-533472dea182" (UID: "ab42a280-8ba3-4a41-bec2-533472dea182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:50.969261 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:50.969231    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p" pod "ab42a280-8ba3-4a41-bec2-533472dea182" (UID: "ab42a280-8ba3-4a41-bec2-533472dea182"). InnerVolumeSpecName "kube-api-access-7pd4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:51.061848 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.061818    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"]
Oct 31 02:07:51.064216 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.064195    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f86e453-57zvs"]
Oct 31 02:07:51.069464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.069420    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:51.069464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.069441    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pd4p\" (UniqueName: \"kubernetes.io/projected/ab42a280-8ba3-4a41-bec2-533472dea182-kube-api-access-7pd4p\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:51.069464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.069453    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab42a280-8ba3-4a41-bec2-533472dea182-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:51.823920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.823894    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"
Oct 31 02:07:51.824081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.823942    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n" event={"ID":"ab42a280-8ba3-4a41-bec2-533472dea182","Type":"ContainerDied","Data":"c1c8e6ed2d1241c5e31e1e1fb566acdd116056a9c86d1cf16c51d0a4ebb18108"}
Oct 31 02:07:51.824081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:51.823990    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1c8e6ed2d1241c5e31e1e1fb566acdd116056a9c86d1cf16c51d0a4ebb18108"
Oct 31 02:07:52.062454 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.062417    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"]
Oct 31 02:07:52.062985 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.062958    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" containerName="revoke-keys"
Oct 31 02:07:52.062985 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.062981    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" containerName="revoke-keys"
Oct 31 02:07:52.063231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.063036    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ab42a280-8ba3-4a41-bec2-533472dea182" containerName="revoke-keys"
Oct 31 02:07:52.063231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.063045    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab42a280-8ba3-4a41-bec2-533472dea182" containerName="revoke-keys"
Oct 31 02:07:52.063231 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.063229    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" containerName="revoke-keys"
Oct 31 02:07:52.063393 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.063252    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab42a280-8ba3-4a41-bec2-533472dea182" containerName="revoke-keys"
Oct 31 02:07:52.066552 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.066534    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.080933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.080866    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tj8k\" (UniqueName: \"kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.081028 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.080932    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.081129 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.081107    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.124572 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.124545    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"]
Oct 31 02:07:52.138703 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.138667    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:07:52.138703 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.138672    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:07:52.138995 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.138711    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:07:52.138995 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.138682    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:07:52.182124 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.182091    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.182343 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.182291    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.182456 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.182353    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tj8k\" (UniqueName: \"kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.183340 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.183313    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.184692 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.184674    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.240590 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.240557    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tj8k\" (UniqueName: \"kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k\") pod \"maas-api-revoke-sub-69e80696-5x79q\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.379413 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.379377    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:52.425156 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.425071    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 02:07:52.486073 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486046    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486271 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486158    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486271 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486192    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486271 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486213    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486472 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486323    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486472 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486350    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486472 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486377    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486472 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486460    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486492    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token\") pod \"689fb428-fba4-418f-b221-a7498437f213\" (UID: \"689fb428-fba4-418f-b221-a7498437f213\") "
Oct 31 02:07:52.486688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486613    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:52.486688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486621    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:52.486850 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486815    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:52.487021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.486995    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:52.487122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.487007    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.487122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.487049    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/689fb428-fba4-418f-b221-a7498437f213-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.487122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.487067    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.487326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.487241    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:52.488884 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.488855    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "kube-api-access-tb6xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:52.489102 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.489081    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:52.489458 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.489431    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:07:52.489458 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.489436    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo" pod "689fb428-fba4-418f-b221-a7498437f213" (UID: "689fb428-fba4-418f-b221-a7498437f213"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:07:52.588302 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.588275    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/689fb428-fba4-418f-b221-a7498437f213-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.588302 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.588303    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.588517 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.588315    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.588517 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.588327    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.588517 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.588341    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/689fb428-fba4-418f-b221-a7498437f213-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.588517 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.588377    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6xc\" (UniqueName: \"kubernetes.io/projected/689fb428-fba4-418f-b221-a7498437f213-kube-api-access-tb6xc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:52.755939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.755910    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"]
Oct 31 02:07:52.757059 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:07:52.757034    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df90395_e58f_4c87_b2c1_b7f9201720e1.slice/crio-ff15f709c9564aa64aa7c797e7f3808793bb0abc37a226cdfdaba1cb11d7d57c WatchSource:0}: Error finding container ff15f709c9564aa64aa7c797e7f3808793bb0abc37a226cdfdaba1cb11d7d57c: Status 404 returned error can't find the container with id ff15f709c9564aa64aa7c797e7f3808793bb0abc37a226cdfdaba1cb11d7d57c
Oct 31 02:07:52.832081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.832047    5419 generic.go:356] "Generic (PLEG): container finished" podID="689fb428-fba4-418f-b221-a7498437f213" containerID="81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2" exitCode=0
Oct 31 02:07:52.832280 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.832184    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"
Oct 31 02:07:52.838068 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.838045    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829d52df-509a-4fa5-b396-b5c00ffb249b" path="/var/lib/kubelet/pods/829d52df-509a-4fa5-b396-b5c00ffb249b/volumes"
Oct 31 02:07:52.838523 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.838502    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" event={"ID":"689fb428-fba4-418f-b221-a7498437f213","Type":"ContainerDied","Data":"81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2"}
Oct 31 02:07:52.838634 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.838540    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk" event={"ID":"689fb428-fba4-418f-b221-a7498437f213","Type":"ContainerDied","Data":"70b49efa106946f4376bbfe1e42344c9f69f52e9244c0913a1e23372831525db"}
Oct 31 02:07:52.838634 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.838559    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q" event={"ID":"2df90395-e58f-4c87-b2c1-b7f9201720e1","Type":"ContainerStarted","Data":"ff15f709c9564aa64aa7c797e7f3808793bb0abc37a226cdfdaba1cb11d7d57c"}
Oct 31 02:07:52.838634 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.838580    5419 scope.go:122] "RemoveContainer" containerID="81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2"
Oct 31 02:07:52.848609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.848591    5419 scope.go:122] "RemoveContainer" containerID="b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"
Oct 31 02:07:52.857638 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.857621    5419 scope.go:122] "RemoveContainer" containerID="81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2"
Oct 31 02:07:52.857909 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:52.857889    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2\": container with ID starting with 81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2 not found: ID does not exist" containerID="81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2"
Oct 31 02:07:52.857959 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.857919    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2"} err="failed to get container status \"81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2\": rpc error: code = NotFound desc = could not find container \"81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2\": container with ID starting with 81dc38bf26afac83ae983efe9545cb5741d251544fe114071e8a0f14340768e2 not found: ID does not exist"
Oct 31 02:07:52.857959 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.857938    5419 scope.go:122] "RemoveContainer" containerID="b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"
Oct 31 02:07:52.858183 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:07:52.858167    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d\": container with ID starting with b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d not found: ID does not exist" containerID="b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"
Oct 31 02:07:52.858270 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.858188    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d"} err="failed to get container status \"b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d\": rpc error: code = NotFound desc = could not find container \"b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d\": container with ID starting with b04ff8635cca8d7d71a7089e3f2f76650de7b92a4dd9eaa2dced506d9bbc186d not found: ID does not exist"
Oct 31 02:07:52.994562 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.994530    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"]
Oct 31 02:07:52.999914 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:52.999890    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-051514-openshift-default-844c7c8789-wb6bk"]
Oct 31 02:07:53.843014 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:53.842984    5419 generic.go:356] "Generic (PLEG): container finished" podID="2df90395-e58f-4c87-b2c1-b7f9201720e1" containerID="9a0288b3bcfbb8a9c4f60b358f04b17a08f13d39a468e40f224f5d1f14d8bc35" exitCode=0
Oct 31 02:07:53.843418 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:53.843032    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q" event={"ID":"2df90395-e58f-4c87-b2c1-b7f9201720e1","Type":"ContainerDied","Data":"9a0288b3bcfbb8a9c4f60b358f04b17a08f13d39a468e40f224f5d1f14d8bc35"}
Oct 31 02:07:54.837849 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:54.837819    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="689fb428-fba4-418f-b221-a7498437f213" path="/var/lib/kubelet/pods/689fb428-fba4-418f-b221-a7498437f213/volumes"
Oct 31 02:07:54.990652 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:54.990627    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:55.014290 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.014266    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca\") pod \"2df90395-e58f-4c87-b2c1-b7f9201720e1\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") "
Oct 31 02:07:55.014420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.014320    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca\") pod \"2df90395-e58f-4c87-b2c1-b7f9201720e1\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") "
Oct 31 02:07:55.014420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.014381    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k\" (UniqueName: \"kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k\") pod \"2df90395-e58f-4c87-b2c1-b7f9201720e1\" (UID: \"2df90395-e58f-4c87-b2c1-b7f9201720e1\") "
Oct 31 02:07:55.014689 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.014663    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca" pod "2df90395-e58f-4c87-b2c1-b7f9201720e1" (UID: "2df90395-e58f-4c87-b2c1-b7f9201720e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:07:55.016571 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.016548    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k" pod "2df90395-e58f-4c87-b2c1-b7f9201720e1" (UID: "2df90395-e58f-4c87-b2c1-b7f9201720e1"). InnerVolumeSpecName "kube-api-access-8tj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:07:55.016664 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.016625    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca" pod "2df90395-e58f-4c87-b2c1-b7f9201720e1" (UID: "2df90395-e58f-4c87-b2c1-b7f9201720e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:07:55.116086 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.116005    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:55.116086 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.116033    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df90395-e58f-4c87-b2c1-b7f9201720e1-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:55.116086 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.116042    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tj8k\" (UniqueName: \"kubernetes.io/projected/2df90395-e58f-4c87-b2c1-b7f9201720e1-kube-api-access-8tj8k\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:07:55.855901 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.855871    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"
Oct 31 02:07:55.855901 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.855897    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q" event={"ID":"2df90395-e58f-4c87-b2c1-b7f9201720e1","Type":"ContainerDied","Data":"ff15f709c9564aa64aa7c797e7f3808793bb0abc37a226cdfdaba1cb11d7d57c"}
Oct 31 02:07:55.856122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:55.855939    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff15f709c9564aa64aa7c797e7f3808793bb0abc37a226cdfdaba1cb11d7d57c"
Oct 31 02:07:57.412381 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:57.412340    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:57.412381 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:57.412353    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:07:58.891073 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:58.891043    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:07:59.555420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:59.555389    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"]
Oct 31 02:07:59.659809 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:07:59.659777    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"]
Oct 31 02:08:01.387064 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:01.387029    5419 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055500fd_ba69_46f9_9824_2ebf5ee9f330.slice/crio-conmon-26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 02:08:01.500407 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.500377    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 02:08:01.563528 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.563503    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 02:08:01.616403 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.616370    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume\") pod \"19e8a15c-fa4c-43ef-a202-3591dd118181\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") "
Oct 31 02:08:01.616582 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.616529    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb\" (UniqueName: \"kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb\") pod \"19e8a15c-fa4c-43ef-a202-3591dd118181\" (UID: \"19e8a15c-fa4c-43ef-a202-3591dd118181\") "
Oct 31 02:08:01.616827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.616801    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume" pod "19e8a15c-fa4c-43ef-a202-3591dd118181" (UID: "19e8a15c-fa4c-43ef-a202-3591dd118181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:01.618800 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.618776    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb" pod "19e8a15c-fa4c-43ef-a202-3591dd118181" (UID: "19e8a15c-fa4c-43ef-a202-3591dd118181"). InnerVolumeSpecName "kube-api-access-5sdcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:01.717774 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.717692    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv\") pod \"055500fd-ba69-46f9-9824-2ebf5ee9f330\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") "
Oct 31 02:08:01.717973 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.717843    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume\") pod \"055500fd-ba69-46f9-9824-2ebf5ee9f330\" (UID: \"055500fd-ba69-46f9-9824-2ebf5ee9f330\") "
Oct 31 02:08:01.718258 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.718241    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e8a15c-fa4c-43ef-a202-3591dd118181-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.718364 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.718262    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdcb\" (UniqueName: \"kubernetes.io/projected/19e8a15c-fa4c-43ef-a202-3591dd118181-kube-api-access-5sdcb\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.718364 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.718303    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume" pod "055500fd-ba69-46f9-9824-2ebf5ee9f330" (UID: "055500fd-ba69-46f9-9824-2ebf5ee9f330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:01.720361 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.720340    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv" pod "055500fd-ba69-46f9-9824-2ebf5ee9f330" (UID: "055500fd-ba69-46f9-9824-2ebf5ee9f330"). InnerVolumeSpecName "kube-api-access-njlbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:01.819052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.819014    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/055500fd-ba69-46f9-9824-2ebf5ee9f330-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.819052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.819049    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlbv\" (UniqueName: \"kubernetes.io/projected/055500fd-ba69-46f9-9824-2ebf5ee9f330-kube-api-access-njlbv\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:01.900743 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.900707    5419 generic.go:356] "Generic (PLEG): container finished" podID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerID="10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e" exitCode=0
Oct 31 02:08:01.900941 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.900777    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"
Oct 31 02:08:01.900941 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.900802    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" event={"ID":"19e8a15c-fa4c-43ef-a202-3591dd118181","Type":"ContainerDied","Data":"10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e"}
Oct 31 02:08:01.900941 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.900873    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk" event={"ID":"19e8a15c-fa4c-43ef-a202-3591dd118181","Type":"ContainerDied","Data":"b9942fbf8a51153e01c34c2585a5b6ac45b458c4d5be384a4938f49deeee5e00"}
Oct 31 02:08:01.900941 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.900904    5419 scope.go:122] "RemoveContainer" containerID="10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e"
Oct 31 02:08:01.902755 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.902730    5419 generic.go:356] "Generic (PLEG): container finished" podID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerID="26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15" exitCode=0
Oct 31 02:08:01.902984 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.902764    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" event={"ID":"055500fd-ba69-46f9-9824-2ebf5ee9f330","Type":"ContainerDied","Data":"26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15"}
Oct 31 02:08:01.903124 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.903005    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl" event={"ID":"055500fd-ba69-46f9-9824-2ebf5ee9f330","Type":"ContainerDied","Data":"6e040e571c34561988720f325236a07d6fa81d1e042f41ae734bef797851f3a3"}
Oct 31 02:08:01.903124 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.902772    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"
Oct 31 02:08:01.912012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.911991    5419 scope.go:122] "RemoveContainer" containerID="10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e"
Oct 31 02:08:01.912365 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:01.912342    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e\": container with ID starting with 10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e not found: ID does not exist" containerID="10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e"
Oct 31 02:08:01.912464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.912372    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e"} err="failed to get container status \"10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e\": rpc error: code = NotFound desc = could not find container \"10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e\": container with ID starting with 10a87dd038befe67d0657d629f1fd54113f7768e7b48b306569631838e0c056e not found: ID does not exist"
Oct 31 02:08:01.912464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.912388    5419 scope.go:122] "RemoveContainer" containerID="26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15"
Oct 31 02:08:01.924335 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.923326    5419 scope.go:122] "RemoveContainer" containerID="26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15"
Oct 31 02:08:01.924335 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:01.923656    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15\": container with ID starting with 26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15 not found: ID does not exist" containerID="26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15"
Oct 31 02:08:01.924335 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:01.923692    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15"} err="failed to get container status \"26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15\": rpc error: code = NotFound desc = could not find container \"26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15\": container with ID starting with 26c03333641d9ad3bd5e5348d907b4d5845700d1941f159d37c147dd93beea15 not found: ID does not exist"
Oct 31 02:08:02.063154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:02.063112    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"]
Oct 31 02:08:02.066361 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:02.066330    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8f6b5b-5764ddfd9f-fc5pl"]
Oct 31 02:08:02.194194 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:02.194164    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"]
Oct 31 02:08:02.199057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:02.199030    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-8f6b5b-554fbb4c45-tchdk"]
Oct 31 02:08:02.837842 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:02.837809    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" path="/var/lib/kubelet/pods/055500fd-ba69-46f9-9824-2ebf5ee9f330/volumes"
Oct 31 02:08:02.838454 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:02.838432    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" path="/var/lib/kubelet/pods/19e8a15c-fa4c-43ef-a202-3591dd118181/volumes"
Oct 31 02:08:04.555815 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.555757    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv" podUID="7a74f67f-98a6-4ca1-92e1-987bdfe3024f" containerName="payload-processing" containerID="cri-o://0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2" gracePeriod=25
Oct 31 02:08:04.661078 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.661043    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c" podUID="6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" containerName="payload-pre-processing" containerID="cri-o://4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e" gracePeriod=25
Oct 31 02:08:04.828073 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.828042    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 02:08:04.906763 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.906740    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 02:08:04.925912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.925882    5419 generic.go:356] "Generic (PLEG): container finished" podID="6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" containerID="4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e" exitCode=0
Oct 31 02:08:04.926056 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.925931    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c" event={"ID":"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb","Type":"ContainerDied","Data":"4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e"}
Oct 31 02:08:04.926056 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.925984    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c" event={"ID":"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb","Type":"ContainerDied","Data":"ff31234caec0a0c19ef3ab24d012ea4fe1426682729a581f3f0eac191891f13a"}
Oct 31 02:08:04.926056 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.925937    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"
Oct 31 02:08:04.926056 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.926017    5419 scope.go:122] "RemoveContainer" containerID="4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e"
Oct 31 02:08:04.927788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.927761    5419 generic.go:356] "Generic (PLEG): container finished" podID="7a74f67f-98a6-4ca1-92e1-987bdfe3024f" containerID="0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2" exitCode=0
Oct 31 02:08:04.927910 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.927809    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"
Oct 31 02:08:04.927910 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.927837    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv" event={"ID":"7a74f67f-98a6-4ca1-92e1-987bdfe3024f","Type":"ContainerDied","Data":"0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2"}
Oct 31 02:08:04.927910 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.927869    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv" event={"ID":"7a74f67f-98a6-4ca1-92e1-987bdfe3024f","Type":"ContainerDied","Data":"f3de448eb380dcb388fb656b4d862743b5ce570d8c78c017391804b3566c2f4f"}
Oct 31 02:08:04.935678 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.935658    5419 scope.go:122] "RemoveContainer" containerID="4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e"
Oct 31 02:08:04.935947 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:04.935930    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e\": container with ID starting with 4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e not found: ID does not exist" containerID="4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e"
Oct 31 02:08:04.936016 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.935954    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e"} err="failed to get container status \"4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e\": rpc error: code = NotFound desc = could not find container \"4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e\": container with ID starting with 4f30c8a58734526d8bdbc568d6474ae2db6f76952827e58bdb22e79b4bd79b7e not found: ID does not exist"
Oct 31 02:08:04.936016 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.935969    5419 scope.go:122] "RemoveContainer" containerID="0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2"
Oct 31 02:08:04.946520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.946505    5419 scope.go:122] "RemoveContainer" containerID="0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2"
Oct 31 02:08:04.946806 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:04.946789    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2\": container with ID starting with 0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2 not found: ID does not exist" containerID="0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2"
Oct 31 02:08:04.946877 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.946813    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2"} err="failed to get container status \"0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2\": rpc error: code = NotFound desc = could not find container \"0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2\": container with ID starting with 0e7a5eb38eb5e564e6b281ebb5823bb51a0ecd6e22caae379aefbbba2a837ef2 not found: ID does not exist"
Oct 31 02:08:04.952175 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.952129    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume\") pod \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") "
Oct 31 02:08:04.952301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.952289    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9\" (UniqueName: \"kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9\") pod \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\" (UID: \"7a74f67f-98a6-4ca1-92e1-987bdfe3024f\") "
Oct 31 02:08:04.952524 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.952495    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume" pod "7a74f67f-98a6-4ca1-92e1-987bdfe3024f" (UID: "7a74f67f-98a6-4ca1-92e1-987bdfe3024f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:04.953505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.953465    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:04.954747 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:04.954726    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9" pod "7a74f67f-98a6-4ca1-92e1-987bdfe3024f" (UID: "7a74f67f-98a6-4ca1-92e1-987bdfe3024f"). InnerVolumeSpecName "kube-api-access-7t4r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:05.054471 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.054439    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9\" (UniqueName: \"kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9\") pod \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") "
Oct 31 02:08:05.054655 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.054533    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume\") pod \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\" (UID: \"6d9a1127-0e5b-4111-9e6e-3687ca47c6fb\") "
Oct 31 02:08:05.054918 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.054890    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4r9\" (UniqueName: \"kubernetes.io/projected/7a74f67f-98a6-4ca1-92e1-987bdfe3024f-kube-api-access-7t4r9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:05.055038 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.054938    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume" pod "6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" (UID: "6d9a1127-0e5b-4111-9e6e-3687ca47c6fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:05.056703 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.056681    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9" pod "6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" (UID: "6d9a1127-0e5b-4111-9e6e-3687ca47c6fb"). InnerVolumeSpecName "kube-api-access-44vs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:05.156163 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.156053    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vs9\" (UniqueName: \"kubernetes.io/projected/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-kube-api-access-44vs9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:05.156163 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.156085    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:05.350036 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.349989    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"]
Oct 31 02:08:05.353639 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.353609    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c4d438-8c7fbcfbc-r588c"]
Oct 31 02:08:05.461344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.461266    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"]
Oct 31 02:08:05.463604 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:05.463583    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c4d438-54dd98698f-vtppv"]
Oct 31 02:08:06.852920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:06.852876    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" path="/var/lib/kubelet/pods/6d9a1127-0e5b-4111-9e6e-3687ca47c6fb/volumes"
Oct 31 02:08:06.853851 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:06.853687    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a74f67f-98a6-4ca1-92e1-987bdfe3024f" path="/var/lib/kubelet/pods/7a74f67f-98a6-4ca1-92e1-987bdfe3024f/volumes"
Oct 31 02:08:14.582551 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:14.582508    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:08:22.572746 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:22.572714    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"]
Oct 31 02:08:31.123691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:31.123659    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"]
Oct 31 02:08:31.172497 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:31.172464    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-98c748-tr9k7"]
Oct 31 02:08:32.836970 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:32.836935    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa4f77e-412b-4d2b-b907-b0882658a0f6" path="/var/lib/kubelet/pods/4aa4f77e-412b-4d2b-b907-b0882658a0f6/volumes"
Oct 31 02:08:37.592858 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.592811    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" podUID="d92daa41-f0c0-47d1-ae86-7714db897538" containerName="main" containerID="cri-o://534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9" gracePeriod=45
Oct 31 02:08:37.848893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.848826    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:08:37.944402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944357    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm\") pod \"d92daa41-f0c0-47d1-ae86-7714db897538\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") "
Oct 31 02:08:37.944549 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944423    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs\" (UniqueName: \"kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs\") pod \"d92daa41-f0c0-47d1-ae86-7714db897538\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") "
Oct 31 02:08:37.944549 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944468    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home\") pod \"d92daa41-f0c0-47d1-ae86-7714db897538\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") "
Oct 31 02:08:37.944549 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944535    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt\" (UniqueName: \"kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt\") pod \"d92daa41-f0c0-47d1-ae86-7714db897538\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") "
Oct 31 02:08:37.944749 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944576    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache\") pod \"d92daa41-f0c0-47d1-ae86-7714db897538\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") "
Oct 31 02:08:37.944749 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944611    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir\") pod \"d92daa41-f0c0-47d1-ae86-7714db897538\" (UID: \"d92daa41-f0c0-47d1-ae86-7714db897538\") "
Oct 31 02:08:37.944868 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944803    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home" pod "d92daa41-f0c0-47d1-ae86-7714db897538" (UID: "d92daa41-f0c0-47d1-ae86-7714db897538"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:37.944925 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944878    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache" pod "d92daa41-f0c0-47d1-ae86-7714db897538" (UID: "d92daa41-f0c0-47d1-ae86-7714db897538"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:37.944970 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.944939    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir" pod "d92daa41-f0c0-47d1-ae86-7714db897538" (UID: "d92daa41-f0c0-47d1-ae86-7714db897538"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:37.945043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.945028    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:37.945099 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.945046    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:37.947000 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.946977    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt" pod "d92daa41-f0c0-47d1-ae86-7714db897538" (UID: "d92daa41-f0c0-47d1-ae86-7714db897538"). InnerVolumeSpecName "kube-api-access-wrkrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:37.947158 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.947119    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs" pod "d92daa41-f0c0-47d1-ae86-7714db897538" (UID: "d92daa41-f0c0-47d1-ae86-7714db897538"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:37.947158 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:37.947124    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm" pod "d92daa41-f0c0-47d1-ae86-7714db897538" (UID: "d92daa41-f0c0-47d1-ae86-7714db897538"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:38.046419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.046389    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:38.046419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.046413    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d92daa41-f0c0-47d1-ae86-7714db897538-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:38.046419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.046425    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d92daa41-f0c0-47d1-ae86-7714db897538-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:38.046651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.046434    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrkrt\" (UniqueName: \"kubernetes.io/projected/d92daa41-f0c0-47d1-ae86-7714db897538-kube-api-access-wrkrt\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:38.143203 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.143089    5419 generic.go:356] "Generic (PLEG): container finished" podID="d92daa41-f0c0-47d1-ae86-7714db897538" containerID="534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9" exitCode=0
Oct 31 02:08:38.143203 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.143153    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"
Oct 31 02:08:38.143203 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.143167    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" event={"ID":"d92daa41-f0c0-47d1-ae86-7714db897538","Type":"ContainerDied","Data":"534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9"}
Oct 31 02:08:38.143478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.143211    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" event={"ID":"d92daa41-f0c0-47d1-ae86-7714db897538","Type":"ContainerDied","Data":"d96119dbdb996879c7d04c8c2f307ec933e556b3b7b39473ff934b344931d81b"}
Oct 31 02:08:38.143478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.143231    5419 scope.go:122] "RemoveContainer" containerID="534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9"
Oct 31 02:08:38.155304 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.155286    5419 scope.go:122] "RemoveContainer" containerID="534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9"
Oct 31 02:08:38.155630 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:38.155599    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9\": container with ID starting with 534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9 not found: ID does not exist" containerID="534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9"
Oct 31 02:08:38.155722 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.155653    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9"} err="failed to get container status \"534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9\": rpc error: code = NotFound desc = could not find container \"534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9\": container with ID starting with 534ef9eb33792f35c78a9a00f693a49d320ec49e403a771f77319619d32b8aa9 not found: ID does not exist"
Oct 31 02:08:38.307550 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.307519    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"]
Oct 31 02:08:38.311538 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.311511    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9"]
Oct 31 02:08:38.551486 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.551451    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"]
Oct 31 02:08:38.551698 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.551677    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" containerID="cri-o://768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764" gracePeriod=30
Oct 31 02:08:38.769987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.769940    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-9e3b95/auth-test-model-9e3b95-kserve-755d4688fc-w4fl9" podUID="d92daa41-f0c0-47d1-ae86-7714db897538" containerName="main" probeResult="failure" output="Get \"https://10.133.1.34:8000/ready\": context deadline exceeded"
Oct 31 02:08:38.837103 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:38.837027    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92daa41-f0c0-47d1-ae86-7714db897538" path="/var/lib/kubelet/pods/d92daa41-f0c0-47d1-ae86-7714db897538/volumes"
Oct 31 02:08:39.594150 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594116    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"]
Oct 31 02:08:39.594595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594582    5419 cpu_manager.go:414] "containerMap: removing container" podUID="6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" containerName="payload-pre-processing"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594597    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" containerName="payload-pre-processing"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594609    5419 cpu_manager.go:414] "containerMap: removing container" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594614    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594621    5419 cpu_manager.go:414] "containerMap: removing container" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerName="payload-processing"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594626    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerName="payload-processing"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594643    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2df90395-e58f-4c87-b2c1-b7f9201720e1" containerName="revoke-keys"
Oct 31 02:08:39.594647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594647    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df90395-e58f-4c87-b2c1-b7f9201720e1" containerName="revoke-keys"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594668    5419 cpu_manager.go:414] "containerMap: removing container" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerName="payload-pre-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594673    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerName="payload-pre-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594686    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7a74f67f-98a6-4ca1-92e1-987bdfe3024f" containerName="payload-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594691    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a74f67f-98a6-4ca1-92e1-987bdfe3024f" containerName="payload-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594698    5419 cpu_manager.go:414] "containerMap: removing container" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594703    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594714    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d92daa41-f0c0-47d1-ae86-7714db897538" containerName="main"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594718    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92daa41-f0c0-47d1-ae86-7714db897538" containerName="main"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594777    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a74f67f-98a6-4ca1-92e1-987bdfe3024f" containerName="payload-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594784    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594792    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92daa41-f0c0-47d1-ae86-7714db897538" containerName="main"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594797    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9a1127-0e5b-4111-9e6e-3687ca47c6fb" containerName="payload-pre-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594804    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e8a15c-fa4c-43ef-a202-3591dd118181" containerName="payload-processing"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594811    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df90395-e58f-4c87-b2c1-b7f9201720e1" containerName="revoke-keys"
Oct 31 02:08:39.594872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.594819    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="055500fd-ba69-46f9-9824-2ebf5ee9f330" containerName="payload-pre-processing"
Oct 31 02:08:39.599388 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.599368    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.654897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.654869    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"]
Oct 31 02:08:39.671083 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.671054    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5zz\" (UniqueName: \"kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.671264 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.671117    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.671264 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.671253    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.700749 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.700725    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:08:39.701090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.701072    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:08:39.701242 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.701100    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:08:39.701242 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.701116    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:08:39.772200 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.772164    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5zz\" (UniqueName: \"kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.772680 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.772241    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.772680 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.772286    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.772891 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.772874    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.774789 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.774765    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.828344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.828315    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5zz\" (UniqueName: \"kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz\") pod \"maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:39.913354 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:39.913280    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:40.243994 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:40.243960    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:08:40.244187 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:40.244019    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:08:40.294980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:40.294948    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"]
Oct 31 02:08:41.164081 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:41.164053    5419 generic.go:356] "Generic (PLEG): container finished" podID="d0175d9a-1e2b-427d-adfb-335128abd607" containerID="65bd083fe2d92b61450bc3b95e44201b7790bdf48363add136cc103926129045" exitCode=0
Oct 31 02:08:41.164489 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:41.164191    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x" event={"ID":"d0175d9a-1e2b-427d-adfb-335128abd607","Type":"ContainerDied","Data":"65bd083fe2d92b61450bc3b95e44201b7790bdf48363add136cc103926129045"}
Oct 31 02:08:41.164489 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:41.164256    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x" event={"ID":"d0175d9a-1e2b-427d-adfb-335128abd607","Type":"ContainerStarted","Data":"f1fd4e770d57abd3c82e0c8834b61dd7fa2f86b59c66a37eb8250a35af67cd38"}
Oct 31 02:08:41.889361 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:41.889326    5419 scope.go:122] "RemoveContainer" containerID="798b32334e198f0eeffaa7f82c24e831921ab5942020845ac9b80b6dada154b1"
Oct 31 02:08:42.302392 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.302367    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:42.411392 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.411357    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca\") pod \"d0175d9a-1e2b-427d-adfb-335128abd607\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") "
Oct 31 02:08:42.411392 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.411397    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca\") pod \"d0175d9a-1e2b-427d-adfb-335128abd607\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") "
Oct 31 02:08:42.411731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.411438    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz\" (UniqueName: \"kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz\") pod \"d0175d9a-1e2b-427d-adfb-335128abd607\" (UID: \"d0175d9a-1e2b-427d-adfb-335128abd607\") "
Oct 31 02:08:42.411843 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.411820    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca" pod "d0175d9a-1e2b-427d-adfb-335128abd607" (UID: "d0175d9a-1e2b-427d-adfb-335128abd607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:42.413734 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.413707    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz" pod "d0175d9a-1e2b-427d-adfb-335128abd607" (UID: "d0175d9a-1e2b-427d-adfb-335128abd607"). InnerVolumeSpecName "kube-api-access-pp5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:42.413829 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.413772    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca" pod "d0175d9a-1e2b-427d-adfb-335128abd607" (UID: "d0175d9a-1e2b-427d-adfb-335128abd607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:42.515078 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.514674    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:42.515078 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.514716    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0175d9a-1e2b-427d-adfb-335128abd607-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:42.515078 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.514732    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5zz\" (UniqueName: \"kubernetes.io/projected/d0175d9a-1e2b-427d-adfb-335128abd607-kube-api-access-pp5zz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:42.785596 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.785558    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"]
Oct 31 02:08:42.787778 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.787743    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8f6b5b-9k6tj"]
Oct 31 02:08:42.837916 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:42.837882    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea71697-c84e-4b6f-a129-c73ef1c79282" path="/var/lib/kubelet/pods/cea71697-c84e-4b6f-a129-c73ef1c79282/volumes"
Oct 31 02:08:43.178043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.177963    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"
Oct 31 02:08:43.178043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.177982    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x" event={"ID":"d0175d9a-1e2b-427d-adfb-335128abd607","Type":"ContainerDied","Data":"f1fd4e770d57abd3c82e0c8834b61dd7fa2f86b59c66a37eb8250a35af67cd38"}
Oct 31 02:08:43.178043 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.178026    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1fd4e770d57abd3c82e0c8834b61dd7fa2f86b59c66a37eb8250a35af67cd38"
Oct 31 02:08:43.812288 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.812262    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 02:08:43.929171 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.928919    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.929588 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929319    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:43.929751 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929555    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.929751 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929690    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.929866 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929754    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.929866 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929791    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.929866 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929839    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.930092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929899    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.930092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.929943    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.930092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.930004    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo\") pod \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\" (UID: \"c8f89ea9-39c8-4263-92e4-34417a61dc0a\") "
Oct 31 02:08:43.930290 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.930118    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:43.931574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.930685    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:43.931574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.930715    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:43.931574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.931017    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:43.931574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.931474    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:43.931574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.931552    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:43.932651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.932626    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "kube-api-access-z5dmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:43.932768 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.932697    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:08:43.933583 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.933562    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:08:43.937273 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:43.937249    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token" pod "c8f89ea9-39c8-4263-92e4-34417a61dc0a" (UID: "c8f89ea9-39c8-4263-92e4-34417a61dc0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:44.031585 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031557    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dmz\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-kube-api-access-z5dmz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.031585 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031580    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.031585 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031592    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.031824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031601    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.031824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031610    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.031824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031618    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f89ea9-39c8-4263-92e4-34417a61dc0a-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.031824 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.031626    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f89ea9-39c8-4263-92e4-34417a61dc0a-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:44.186442 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.186346    5419 generic.go:356] "Generic (PLEG): container finished" podID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerID="768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764" exitCode=0
Oct 31 02:08:44.186442 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.186398    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"
Oct 31 02:08:44.186442 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.186434    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" event={"ID":"c8f89ea9-39c8-4263-92e4-34417a61dc0a","Type":"ContainerDied","Data":"768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764"}
Oct 31 02:08:44.186684 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.186492    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt" event={"ID":"c8f89ea9-39c8-4263-92e4-34417a61dc0a","Type":"ContainerDied","Data":"b2acfa9cb2f6603fb522d1ff3a5db1b5b1d559896f5e81b0a6f02df8042a104a"}
Oct 31 02:08:44.186684 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.186516    5419 scope.go:122] "RemoveContainer" containerID="768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764"
Oct 31 02:08:44.200662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.200642    5419 scope.go:122] "RemoveContainer" containerID="c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"
Oct 31 02:08:44.212299 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.212237    5419 scope.go:122] "RemoveContainer" containerID="768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764"
Oct 31 02:08:44.212574 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:44.212551    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764\": container with ID starting with 768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764 not found: ID does not exist" containerID="768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764"
Oct 31 02:08:44.212626 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.212584    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764"} err="failed to get container status \"768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764\": rpc error: code = NotFound desc = could not find container \"768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764\": container with ID starting with 768e7b3cef300f0e36d7c5e615af4fbd4902cbc771eacbf72ab1b3cc5dff6764 not found: ID does not exist"
Oct 31 02:08:44.212626 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.212603    5419 scope.go:122] "RemoveContainer" containerID="c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"
Oct 31 02:08:44.212863 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:44.212844    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811\": container with ID starting with c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811 not found: ID does not exist" containerID="c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"
Oct 31 02:08:44.212907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.212870    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811"} err="failed to get container status \"c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811\": rpc error: code = NotFound desc = could not find container \"c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811\": container with ID starting with c60f8437cad34e8390576e1d047dc4a2ea2067f0ac313a995342e32c589d4811 not found: ID does not exist"
Oct 31 02:08:44.354297 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.354193    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"]
Oct 31 02:08:44.402914 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.402880    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-98c748-openshift-default-5758c954bc-zd4mt"]
Oct 31 02:08:44.583501 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:08:44.583463    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:08:44.845283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:44.845197    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" path="/var/lib/kubelet/pods/c8f89ea9-39c8-4263-92e4-34417a61dc0a/volumes"
Oct 31 02:08:51.075355 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:51.075321    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"]
Oct 31 02:08:51.179173 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:51.179123    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"]
Oct 31 02:08:53.907485 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:53.907451    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"]
Oct 31 02:08:53.956553 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:53.956523    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c4d438-8r42n"]
Oct 31 02:08:54.377565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.377531    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"]
Oct 31 02:08:54.377997 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.377984    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy"
Oct 31 02:08:54.378048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.377999    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy"
Oct 31 02:08:54.378048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.378028    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d0175d9a-1e2b-427d-adfb-335128abd607" containerName="revoke-keys"
Oct 31 02:08:54.378048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.378034    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0175d9a-1e2b-427d-adfb-335128abd607" containerName="revoke-keys"
Oct 31 02:08:54.378185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.378106    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy"
Oct 31 02:08:54.378185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.378117    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="689fb428-fba4-418f-b221-a7498437f213" containerName="istio-proxy"
Oct 31 02:08:54.378185 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.378125    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0175d9a-1e2b-427d-adfb-335128abd607" containerName="revoke-keys"
Oct 31 02:08:54.382662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.382646    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.435153 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.435042    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:08:54.435297 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.435180    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:08:54.438354 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.438325    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"]
Oct 31 02:08:54.484507 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.484483    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:08:54.484507 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.484493    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:08:54.542895 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.542868    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.543030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.542922    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfjs\" (UniqueName: \"kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.543030 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.542998    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.644401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.644327    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.644401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.644383    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfjs\" (UniqueName: \"kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.644595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.644413    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.644981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.644959    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.646973 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.646946    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.700665 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.700634    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfjs\" (UniqueName: \"kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:54.836459 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.836428    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab42a280-8ba3-4a41-bec2-533472dea182" path="/var/lib/kubelet/pods/ab42a280-8ba3-4a41-bec2-533472dea182/volumes"
Oct 31 02:08:54.995403 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:54.995375    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:55.376506 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:55.376481    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"]
Oct 31 02:08:55.378268 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:08:55.378242    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ce8675_d4c6_4cd6_ad43_26db7c37bc3a.slice/crio-43afbdc112f1b1e1d980b4268ff52806ab9f6f7d5d8e8e6d2e0866922cbeeaac WatchSource:0}: Error finding container 43afbdc112f1b1e1d980b4268ff52806ab9f6f7d5d8e8e6d2e0866922cbeeaac: Status 404 returned error can't find the container with id 43afbdc112f1b1e1d980b4268ff52806ab9f6f7d5d8e8e6d2e0866922cbeeaac
Oct 31 02:08:56.076203 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:56.076160    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerName="payload-processing" containerID="cri-o://9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688" gracePeriod=25
Oct 31 02:08:56.180088 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:56.180044    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerName="payload-pre-processing" containerID="cri-o://e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f" gracePeriod=25
Oct 31 02:08:56.265005 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:56.264974    5419 generic.go:356] "Generic (PLEG): container finished" podID="10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" containerID="f74cdce3a690ad2e13de80bde9c36f109d1493d5486845e8941a5f3212736aca" exitCode=0
Oct 31 02:08:56.265154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:56.265051    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d" event={"ID":"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a","Type":"ContainerDied","Data":"f74cdce3a690ad2e13de80bde9c36f109d1493d5486845e8941a5f3212736aca"}
Oct 31 02:08:56.265154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:56.265125    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d" event={"ID":"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a","Type":"ContainerStarted","Data":"43afbdc112f1b1e1d980b4268ff52806ab9f6f7d5d8e8e6d2e0866922cbeeaac"}
Oct 31 02:08:57.412612 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.412590    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:57.574848 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.574773    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca\") pod \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") "
Oct 31 02:08:57.574848 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.574814    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca\") pod \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") "
Oct 31 02:08:57.574848 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.574835    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs\" (UniqueName: \"kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs\") pod \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\" (UID: \"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a\") "
Oct 31 02:08:57.575183 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.575129    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca" pod "10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" (UID: "10ce8675-d4c6-4cd6-ad43-26db7c37bc3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:08:57.575260 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.575239    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:57.577182 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.577160    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca" pod "10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" (UID: "10ce8675-d4c6-4cd6-ad43-26db7c37bc3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:08:57.577291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.577249    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs" pod "10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" (UID: "10ce8675-d4c6-4cd6-ad43-26db7c37bc3a"). InnerVolumeSpecName "kube-api-access-kdfjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:08:57.675996 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.675968    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:57.675996 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:57.675993    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdfjs\" (UniqueName: \"kubernetes.io/projected/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a-kube-api-access-kdfjs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:08:58.278742 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.278708    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"
Oct 31 02:08:58.278939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.278743    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d" event={"ID":"10ce8675-d4c6-4cd6-ad43-26db7c37bc3a","Type":"ContainerDied","Data":"43afbdc112f1b1e1d980b4268ff52806ab9f6f7d5d8e8e6d2e0866922cbeeaac"}
Oct 31 02:08:58.278939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.278800    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43afbdc112f1b1e1d980b4268ff52806ab9f6f7d5d8e8e6d2e0866922cbeeaac"
Oct 31 02:08:58.712025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.711949    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"]
Oct 31 02:08:58.712510 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.712494    5419 cpu_manager.go:414] "containerMap: removing container" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy"
Oct 31 02:08:58.712559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.712512    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy"
Oct 31 02:08:58.712559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.712525    5419 cpu_manager.go:414] "containerMap: removing container" podUID="10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" containerName="revoke-keys"
Oct 31 02:08:58.712559 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.712530    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" containerName="revoke-keys"
Oct 31 02:08:58.712660 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.712607    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f89ea9-39c8-4263-92e4-34417a61dc0a" containerName="istio-proxy"
Oct 31 02:08:58.712660 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.712617    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" containerName="revoke-keys"
Oct 31 02:08:58.715943 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.715926    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.769276 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.769250    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:08:58.769537 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.769495    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:08:58.769537 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.769536    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:08:58.769708 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.769557    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:08:58.772199 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.772174    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"]
Oct 31 02:08:58.897738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.897705    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcjp\" (UniqueName: \"kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.897922 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.897860    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.897922 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.897898    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.998700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.998671    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcjp\" (UniqueName: \"kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.998876 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.998769    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.998876 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.998798    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:58.999501 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:58.999476    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:59.001337 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:59.001316    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:59.055279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:59.055255    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcjp\" (UniqueName: \"kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp\") pod \"maas-api-revoke-keys-e2e-autores-183266-9hjh2\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:59.329683 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:59.329596    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:08:59.486464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:59.486420    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:08:59.486630 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:59.486422    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:08:59.711321 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:08:59.711292    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"]
Oct 31 02:08:59.712673 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:08:59.712650    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b7e1bf_2209_4a47_94cf_af8170430e88.slice/crio-16523d3ee75307d444659d5d93b4c7912531e700eab1beade997b7f69bf9fe18 WatchSource:0}: Error finding container 16523d3ee75307d444659d5d93b4c7912531e700eab1beade997b7f69bf9fe18: Status 404 returned error can't find the container with id 16523d3ee75307d444659d5d93b4c7912531e700eab1beade997b7f69bf9fe18
Oct 31 02:09:00.292517 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:00.292482    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2" event={"ID":"71b7e1bf-2209-4a47-94cf-af8170430e88","Type":"ContainerStarted","Data":"2a8b1adc282f99346a5c118ee2db3d3bd6b47a7e2cc1ce1b056cd9085d8419cc"}
Oct 31 02:09:00.292690 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:00.292542    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2" event={"ID":"71b7e1bf-2209-4a47-94cf-af8170430e88","Type":"ContainerStarted","Data":"16523d3ee75307d444659d5d93b4c7912531e700eab1beade997b7f69bf9fe18"}
Oct 31 02:09:00.457275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:00.457160    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2" podStartSLOduration=2.45713152 podStartE2EDuration="2.45713152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:00.406240917 +0000 UTC m=+3861.707197495" watchObservedRunningTime="2026-10-31 02:09:00.45713152 +0000 UTC m=+3861.758088097"
Oct 31 02:09:01.313130 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:01.313089    5419 generic.go:356] "Generic (PLEG): container finished" podID="71b7e1bf-2209-4a47-94cf-af8170430e88" containerID="2a8b1adc282f99346a5c118ee2db3d3bd6b47a7e2cc1ce1b056cd9085d8419cc" exitCode=0
Oct 31 02:09:01.313608 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:01.313177    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2" event={"ID":"71b7e1bf-2209-4a47-94cf-af8170430e88","Type":"ContainerDied","Data":"2a8b1adc282f99346a5c118ee2db3d3bd6b47a7e2cc1ce1b056cd9085d8419cc"}
Oct 31 02:09:02.464348 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.464323    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:09:02.542491 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.542463    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca\") pod \"71b7e1bf-2209-4a47-94cf-af8170430e88\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") "
Oct 31 02:09:02.542637 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.542508    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp\" (UniqueName: \"kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp\") pod \"71b7e1bf-2209-4a47-94cf-af8170430e88\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") "
Oct 31 02:09:02.542637 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.542529    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca\") pod \"71b7e1bf-2209-4a47-94cf-af8170430e88\" (UID: \"71b7e1bf-2209-4a47-94cf-af8170430e88\") "
Oct 31 02:09:02.542904 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.542881    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca" pod "71b7e1bf-2209-4a47-94cf-af8170430e88" (UID: "71b7e1bf-2209-4a47-94cf-af8170430e88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:02.544793 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.544768    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca" pod "71b7e1bf-2209-4a47-94cf-af8170430e88" (UID: "71b7e1bf-2209-4a47-94cf-af8170430e88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:02.544904 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.544797    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp" pod "71b7e1bf-2209-4a47-94cf-af8170430e88" (UID: "71b7e1bf-2209-4a47-94cf-af8170430e88"). InnerVolumeSpecName "kube-api-access-2qcjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:02.644208 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.644109    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:02.644208 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.644163    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcjp\" (UniqueName: \"kubernetes.io/projected/71b7e1bf-2209-4a47-94cf-af8170430e88-kube-api-access-2qcjp\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:02.644208 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:02.644176    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b7e1bf-2209-4a47-94cf-af8170430e88-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:03.326569 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:03.326536    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"
Oct 31 02:09:03.326727 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:03.326566    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2" event={"ID":"71b7e1bf-2209-4a47-94cf-af8170430e88","Type":"ContainerDied","Data":"16523d3ee75307d444659d5d93b4c7912531e700eab1beade997b7f69bf9fe18"}
Oct 31 02:09:03.326727 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:03.326621    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16523d3ee75307d444659d5d93b4c7912531e700eab1beade997b7f69bf9fe18"
Oct 31 02:09:05.855764 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:05.855728    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"]
Oct 31 02:09:05.958998 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:05.958962    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"]
Oct 31 02:09:06.551583 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:06.551547    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:09:09.486528 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:09.486497    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:09.486910 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:09.486543    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:09:10.856057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:10.856021    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq" podUID="0b80986f-569d-4cf1-b128-b01992b59b59" containerName="payload-processing" containerID="cri-o://e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f" gracePeriod=25
Oct 31 02:09:10.961797 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:10.961723    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4" podUID="478fa64f-fb3c-42ed-b828-e084c661362a" containerName="payload-pre-processing" containerID="cri-o://9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d" gracePeriod=25
Oct 31 02:09:11.144438 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.144412    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 02:09:11.253059 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.253034    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 02:09:11.267696 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.267662    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5\" (UniqueName: \"kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5\") pod \"0b80986f-569d-4cf1-b128-b01992b59b59\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") "
Oct 31 02:09:11.267873 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.267739    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume\") pod \"0b80986f-569d-4cf1-b128-b01992b59b59\" (UID: \"0b80986f-569d-4cf1-b128-b01992b59b59\") "
Oct 31 02:09:11.268159 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.268121    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume" pod "0b80986f-569d-4cf1-b128-b01992b59b59" (UID: "0b80986f-569d-4cf1-b128-b01992b59b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:11.270244 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.270215    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5" pod "0b80986f-569d-4cf1-b128-b01992b59b59" (UID: "0b80986f-569d-4cf1-b128-b01992b59b59"). InnerVolumeSpecName "kube-api-access-gkhd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:11.368759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.368730    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume\") pod \"478fa64f-fb3c-42ed-b828-e084c661362a\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") "
Oct 31 02:09:11.368963 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.368775    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth\" (UniqueName: \"kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth\") pod \"478fa64f-fb3c-42ed-b828-e084c661362a\" (UID: \"478fa64f-fb3c-42ed-b828-e084c661362a\") "
Oct 31 02:09:11.369218 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.369189    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume" pod "478fa64f-fb3c-42ed-b828-e084c661362a" (UID: "478fa64f-fb3c-42ed-b828-e084c661362a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:11.369328 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.369196    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkhd5\" (UniqueName: \"kubernetes.io/projected/0b80986f-569d-4cf1-b128-b01992b59b59-kube-api-access-gkhd5\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:11.369328 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.369252    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b80986f-569d-4cf1-b128-b01992b59b59-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:11.371260 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.371155    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth" pod "478fa64f-fb3c-42ed-b828-e084c661362a" (UID: "478fa64f-fb3c-42ed-b828-e084c661362a"). InnerVolumeSpecName "kube-api-access-zjlth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:11.377478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.377456    5419 generic.go:356] "Generic (PLEG): container finished" podID="0b80986f-569d-4cf1-b128-b01992b59b59" containerID="e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f" exitCode=0
Oct 31 02:09:11.377584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.377517    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"
Oct 31 02:09:11.377584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.377536    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq" event={"ID":"0b80986f-569d-4cf1-b128-b01992b59b59","Type":"ContainerDied","Data":"e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f"}
Oct 31 02:09:11.377688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.377589    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq" event={"ID":"0b80986f-569d-4cf1-b128-b01992b59b59","Type":"ContainerDied","Data":"857656d7f00c84d6e7466f5f0ae651b4a295dd453db53096c913c0e80772efdd"}
Oct 31 02:09:11.377688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.377621    5419 scope.go:122] "RemoveContainer" containerID="e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f"
Oct 31 02:09:11.379041 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.379020    5419 generic.go:356] "Generic (PLEG): container finished" podID="478fa64f-fb3c-42ed-b828-e084c661362a" containerID="9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d" exitCode=0
Oct 31 02:09:11.379197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.379056    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4" event={"ID":"478fa64f-fb3c-42ed-b828-e084c661362a","Type":"ContainerDied","Data":"9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d"}
Oct 31 02:09:11.379197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.379070    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"
Oct 31 02:09:11.379197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.379095    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4" event={"ID":"478fa64f-fb3c-42ed-b828-e084c661362a","Type":"ContainerDied","Data":"ad55ccddd4f5e933d2ffb81c4f351835c38660c8cb5f0dbcf7c977c5aacb3a38"}
Oct 31 02:09:11.390965 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.390944    5419 scope.go:122] "RemoveContainer" containerID="e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f"
Oct 31 02:09:11.391276 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:11.391257    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f\": container with ID starting with e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f not found: ID does not exist" containerID="e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f"
Oct 31 02:09:11.391358 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.391284    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f"} err="failed to get container status \"e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f\": rpc error: code = NotFound desc = could not find container \"e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f\": container with ID starting with e322fe23ebd3fba55afa0083128c0b44f9f5d51a7c4d2f85ed7b07eca450573f not found: ID does not exist"
Oct 31 02:09:11.391358 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.391302    5419 scope.go:122] "RemoveContainer" containerID="9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d"
Oct 31 02:09:11.402078 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.402057    5419 scope.go:122] "RemoveContainer" containerID="9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d"
Oct 31 02:09:11.402372 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:11.402353    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d\": container with ID starting with 9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d not found: ID does not exist" containerID="9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d"
Oct 31 02:09:11.402429 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.402379    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d"} err="failed to get container status \"9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d\": rpc error: code = NotFound desc = could not find container \"9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d\": container with ID starting with 9edb5b5e4c3099a829606405dec96e0e97d6ff0a590aca16b852f4f48485d83d not found: ID does not exist"
Oct 31 02:09:11.470505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.470473    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478fa64f-fb3c-42ed-b828-e084c661362a-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:11.470505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.470502    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjlth\" (UniqueName: \"kubernetes.io/projected/478fa64f-fb3c-42ed-b828-e084c661362a-kube-api-access-zjlth\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:11.540153 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.540109    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"]
Oct 31 02:09:11.545876 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.545846    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9e3b95-56f5bbddf5-llpqq"]
Oct 31 02:09:11.660247 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.660163    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"]
Oct 31 02:09:11.660494 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:11.660478    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9e3b95-7c86c4d6ff-j4zz4"]
Oct 31 02:09:12.257266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257228    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"]
Oct 31 02:09:12.257756 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257739    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0b80986f-569d-4cf1-b128-b01992b59b59" containerName="payload-processing"
Oct 31 02:09:12.257860 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257760    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b80986f-569d-4cf1-b128-b01992b59b59" containerName="payload-processing"
Oct 31 02:09:12.257860 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257805    5419 cpu_manager.go:414] "containerMap: removing container" podUID="71b7e1bf-2209-4a47-94cf-af8170430e88" containerName="revoke-keys"
Oct 31 02:09:12.257860 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257812    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b7e1bf-2209-4a47-94cf-af8170430e88" containerName="revoke-keys"
Oct 31 02:09:12.257860 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257821    5419 cpu_manager.go:414] "containerMap: removing container" podUID="478fa64f-fb3c-42ed-b828-e084c661362a" containerName="payload-pre-processing"
Oct 31 02:09:12.257860 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257829    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478fa64f-fb3c-42ed-b828-e084c661362a" containerName="payload-pre-processing"
Oct 31 02:09:12.258183 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257906    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b7e1bf-2209-4a47-94cf-af8170430e88" containerName="revoke-keys"
Oct 31 02:09:12.258183 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257914    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="478fa64f-fb3c-42ed-b828-e084c661362a" containerName="payload-pre-processing"
Oct 31 02:09:12.258183 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.257925    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b80986f-569d-4cf1-b128-b01992b59b59" containerName="payload-processing"
Oct 31 02:09:12.262710 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.262688    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.319121 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.319059    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:09:12.319121 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.319104    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:09:12.319423 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.319194    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:09:12.323117 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.323085    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"]
Oct 31 02:09:12.323343 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.323219    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:09:12.379166 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.379112    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvvq\" (UniqueName: \"kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.379387 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.379354    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.379537 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.379420    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.480092 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.480053    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.480342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.480104    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.480342 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.480183    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvvq\" (UniqueName: \"kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.480853 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.480833    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.482616 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.482595    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.536478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.536416    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvvq\" (UniqueName: \"kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq\") pod \"maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.576882 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.576854    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:12.836868 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.836800    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b80986f-569d-4cf1-b128-b01992b59b59" path="/var/lib/kubelet/pods/0b80986f-569d-4cf1-b128-b01992b59b59/volumes"
Oct 31 02:09:12.837215 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.837201    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478fa64f-fb3c-42ed-b828-e084c661362a" path="/var/lib/kubelet/pods/478fa64f-fb3c-42ed-b828-e084c661362a/volumes"
Oct 31 02:09:12.949897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:12.949874    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"]
Oct 31 02:09:12.951793 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:12.951768    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d008513_4ca3_479e_bcf2_6a7d8cb0d932.slice/crio-9de2100bed4b673c38b03fc10ae4d22922fbaabfc0267e8127d8471a7e58aae0 WatchSource:0}: Error finding container 9de2100bed4b673c38b03fc10ae4d22922fbaabfc0267e8127d8471a7e58aae0: Status 404 returned error can't find the container with id 9de2100bed4b673c38b03fc10ae4d22922fbaabfc0267e8127d8471a7e58aae0
Oct 31 02:09:13.394291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:13.394254    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls" event={"ID":"6d008513-4ca3-479e-bcf2-6a7d8cb0d932","Type":"ContainerStarted","Data":"58c66d8bb6c4721f30b026f1386f67091762e5d4d5ece651a07133ad95e2797d"}
Oct 31 02:09:13.394740 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:13.394309    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls" event={"ID":"6d008513-4ca3-479e-bcf2-6a7d8cb0d932","Type":"ContainerStarted","Data":"9de2100bed4b673c38b03fc10ae4d22922fbaabfc0267e8127d8471a7e58aae0"}
Oct 31 02:09:13.508638 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:13.508584    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls" podStartSLOduration=1.508570911 podStartE2EDuration="1.508570911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:13.506578509 +0000 UTC m=+3874.807535086" watchObservedRunningTime="2026-10-31 02:09:13.508570911 +0000 UTC m=+3874.809527488"
Oct 31 02:09:14.401657 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:14.401621    5419 generic.go:356] "Generic (PLEG): container finished" podID="6d008513-4ca3-479e-bcf2-6a7d8cb0d932" containerID="58c66d8bb6c4721f30b026f1386f67091762e5d4d5ece651a07133ad95e2797d" exitCode=0
Oct 31 02:09:14.402128 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:14.401708    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls" event={"ID":"6d008513-4ca3-479e-bcf2-6a7d8cb0d932","Type":"ContainerDied","Data":"58c66d8bb6c4721f30b026f1386f67091762e5d4d5ece651a07133ad95e2797d"}
Oct 31 02:09:14.585943 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:14.585903    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:09:15.547121 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.547098    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:15.714953 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.714868    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca\") pod \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") "
Oct 31 02:09:15.714953 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.714947    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca\") pod \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") "
Oct 31 02:09:15.715256 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.714997    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq\" (UniqueName: \"kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq\") pod \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\" (UID: \"6d008513-4ca3-479e-bcf2-6a7d8cb0d932\") "
Oct 31 02:09:15.715325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.715291    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca" pod "6d008513-4ca3-479e-bcf2-6a7d8cb0d932" (UID: "6d008513-4ca3-479e-bcf2-6a7d8cb0d932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:15.717220 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.717194    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca" pod "6d008513-4ca3-479e-bcf2-6a7d8cb0d932" (UID: "6d008513-4ca3-479e-bcf2-6a7d8cb0d932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:15.717356 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.717335    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq" pod "6d008513-4ca3-479e-bcf2-6a7d8cb0d932" (UID: "6d008513-4ca3-479e-bcf2-6a7d8cb0d932"). InnerVolumeSpecName "kube-api-access-hnvvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:15.816652 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.816615    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:15.816652 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.816643    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:15.816652 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:15.816654    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvvq\" (UniqueName: \"kubernetes.io/projected/6d008513-4ca3-479e-bcf2-6a7d8cb0d932-kube-api-access-hnvvq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.311945 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.311920    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 02:09:16.409600 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.409568    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 02:09:16.415239 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.415215    5419 generic.go:356] "Generic (PLEG): container finished" podID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerID="9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688" exitCode=0
Oct 31 02:09:16.415382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.415261    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"
Oct 31 02:09:16.415382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.415295    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" event={"ID":"43b3b334-2c23-4df0-9410-c4695698a5fd","Type":"ContainerDied","Data":"9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688"}
Oct 31 02:09:16.415478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.415381    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr" event={"ID":"43b3b334-2c23-4df0-9410-c4695698a5fd","Type":"ContainerDied","Data":"3da5f62dbfe0b559aafb2b0fc6f2d6eee67973edff1d36a53bb5b00616a6bc6e"}
Oct 31 02:09:16.415478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.415417    5419 scope.go:122] "RemoveContainer" containerID="9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688"
Oct 31 02:09:16.416698 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.416679    5419 generic.go:356] "Generic (PLEG): container finished" podID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerID="e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f" exitCode=0
Oct 31 02:09:16.416762 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.416717    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"
Oct 31 02:09:16.416817 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.416776    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" event={"ID":"9019937e-21f8-49d3-b2e3-fbfe90eaffdf","Type":"ContainerDied","Data":"e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f"}
Oct 31 02:09:16.417004 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.416989    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4" event={"ID":"9019937e-21f8-49d3-b2e3-fbfe90eaffdf","Type":"ContainerDied","Data":"47df04785de17794b5aa7599caae355b7a334b69175c15ef6e26c56fb3bda9ce"}
Oct 31 02:09:16.418163 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.418117    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"
Oct 31 02:09:16.418253 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.418157    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls" event={"ID":"6d008513-4ca3-479e-bcf2-6a7d8cb0d932","Type":"ContainerDied","Data":"9de2100bed4b673c38b03fc10ae4d22922fbaabfc0267e8127d8471a7e58aae0"}
Oct 31 02:09:16.418253 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.418205    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9de2100bed4b673c38b03fc10ae4d22922fbaabfc0267e8127d8471a7e58aae0"
Oct 31 02:09:16.425013 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.424997    5419 scope.go:122] "RemoveContainer" containerID="9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688"
Oct 31 02:09:16.425334 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:16.425308    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688\": container with ID starting with 9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688 not found: ID does not exist" containerID="9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688"
Oct 31 02:09:16.425387 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.425337    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688"} err="failed to get container status \"9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688\": rpc error: code = NotFound desc = could not find container \"9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688\": container with ID starting with 9bad3747c1ef2e74584bffea902558d916642a102d9a89eeacae72742ec9d688 not found: ID does not exist"
Oct 31 02:09:16.425387 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.425355    5419 scope.go:122] "RemoveContainer" containerID="e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f"
Oct 31 02:09:16.428592 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.428575    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z\" (UniqueName: \"kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z\") pod \"43b3b334-2c23-4df0-9410-c4695698a5fd\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") "
Oct 31 02:09:16.428693 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.428681    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume\") pod \"43b3b334-2c23-4df0-9410-c4695698a5fd\" (UID: \"43b3b334-2c23-4df0-9410-c4695698a5fd\") "
Oct 31 02:09:16.429101 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.429076    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume" pod "43b3b334-2c23-4df0-9410-c4695698a5fd" (UID: "43b3b334-2c23-4df0-9410-c4695698a5fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:16.430867 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.430837    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z" pod "43b3b334-2c23-4df0-9410-c4695698a5fd" (UID: "43b3b334-2c23-4df0-9410-c4695698a5fd"). InnerVolumeSpecName "kube-api-access-8l55z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:16.437290 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.437221    5419 scope.go:122] "RemoveContainer" containerID="e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f"
Oct 31 02:09:16.437538 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:16.437515    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f\": container with ID starting with e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f not found: ID does not exist" containerID="e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f"
Oct 31 02:09:16.437608 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.437549    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f"} err="failed to get container status \"e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f\": rpc error: code = NotFound desc = could not find container \"e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f\": container with ID starting with e5265ae9d41b7211deccd16626b67e2b86abb0ab1bbb02e63690b23f6e9bd75f not found: ID does not exist"
Oct 31 02:09:16.530075 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.529979    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume\") pod \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") "
Oct 31 02:09:16.530306 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.530091    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm\" (UniqueName: \"kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm\") pod \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\" (UID: \"9019937e-21f8-49d3-b2e3-fbfe90eaffdf\") "
Oct 31 02:09:16.530689 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.530423    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume" pod "9019937e-21f8-49d3-b2e3-fbfe90eaffdf" (UID: "9019937e-21f8-49d3-b2e3-fbfe90eaffdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:16.530689 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.530434    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l55z\" (UniqueName: \"kubernetes.io/projected/43b3b334-2c23-4df0-9410-c4695698a5fd-kube-api-access-8l55z\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.530689 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.530458    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b3b334-2c23-4df0-9410-c4695698a5fd-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.532616 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.532591    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm" pod "9019937e-21f8-49d3-b2e3-fbfe90eaffdf" (UID: "9019937e-21f8-49d3-b2e3-fbfe90eaffdf"). InnerVolumeSpecName "kube-api-access-chgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:16.631475 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.631395    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.631475 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.631423    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chgrm\" (UniqueName: \"kubernetes.io/projected/9019937e-21f8-49d3-b2e3-fbfe90eaffdf-kube-api-access-chgrm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:16.683279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.683249    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"]
Oct 31 02:09:16.906196 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.906098    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"]
Oct 31 02:09:16.911620 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:16.911589    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057e2d-694f5f4fb5-d4fmr"]
Oct 31 02:09:17.019651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:17.019611    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"]
Oct 31 02:09:17.021764 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:17.021742    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057e2d-56fdf7f54c-9jjs4"]
Oct 31 02:09:18.403038 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.402956    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"]
Oct 31 02:09:18.403446 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403416    5419 cpu_manager.go:414] "containerMap: removing container" podUID="6d008513-4ca3-479e-bcf2-6a7d8cb0d932" containerName="revoke-keys"
Oct 31 02:09:18.403446 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403428    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d008513-4ca3-479e-bcf2-6a7d8cb0d932" containerName="revoke-keys"
Oct 31 02:09:18.403446 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403446    5419 cpu_manager.go:414] "containerMap: removing container" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerName="payload-processing"
Oct 31 02:09:18.403573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403452    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerName="payload-processing"
Oct 31 02:09:18.403573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403483    5419 cpu_manager.go:414] "containerMap: removing container" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerName="payload-pre-processing"
Oct 31 02:09:18.403573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403488    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerName="payload-pre-processing"
Oct 31 02:09:18.403573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403545    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" containerName="payload-processing"
Oct 31 02:09:18.403573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403559    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" containerName="payload-pre-processing"
Oct 31 02:09:18.403573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.403566    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d008513-4ca3-479e-bcf2-6a7d8cb0d932" containerName="revoke-keys"
Oct 31 02:09:18.408298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.408280    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.460259 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.460219    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:09:18.460259 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.460231    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:09:18.509671 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.509636    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:09:18.509671 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.509660    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:09:18.513284 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.513266    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"]
Oct 31 02:09:18.516924 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.516909    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"]
Oct 31 02:09:18.517002 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.516992    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.550731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.550700    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.550876 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.550747    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9tc\" (UniqueName: \"kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.550938 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.550918    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.563604 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.563567    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"]
Oct 31 02:09:18.651758 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.651724    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.651942 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.651766    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.651942 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.651796    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9tc\" (UniqueName: \"kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.651942 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.651883    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.651942 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.651929    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blj4j\" (UniqueName: \"kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.652225 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.652096    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.652392 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.652375    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.654533 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.654481    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.708144 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.708116    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9tc\" (UniqueName: \"kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc\") pod \"maas-api-revoke-sub-0392c9d6-h28c2\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.721991 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.721973    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:18.753247 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.753213    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blj4j\" (UniqueName: \"kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.753395 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.753344    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.753459 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.753432    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.754106 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.754083    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.755817 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.755796    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.810510 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.810479    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blj4j\" (UniqueName: \"kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j\") pod \"maas-api-revoke-sub-7d7915c6-hx5rl\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.832285 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.832259    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:18.836987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.836962    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b3b334-2c23-4df0-9410-c4695698a5fd" path="/var/lib/kubelet/pods/43b3b334-2c23-4df0-9410-c4695698a5fd/volumes"
Oct 31 02:09:18.837507 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:18.837488    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9019937e-21f8-49d3-b2e3-fbfe90eaffdf" path="/var/lib/kubelet/pods/9019937e-21f8-49d3-b2e3-fbfe90eaffdf/volumes"
Oct 31 02:09:19.094769 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:19.094739    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"]
Oct 31 02:09:19.096405 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:19.096381    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f7be2b_866d_4e43_8ee7_b9d711600a64.slice/crio-f963de6d7dde4640468ee0ab5196af701ce4b2e7e26c350a136b9e56bb6a93f0 WatchSource:0}: Error finding container f963de6d7dde4640468ee0ab5196af701ce4b2e7e26c350a136b9e56bb6a93f0: Status 404 returned error can't find the container with id f963de6d7dde4640468ee0ab5196af701ce4b2e7e26c350a136b9e56bb6a93f0
Oct 31 02:09:19.203113 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:19.203086    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"]
Oct 31 02:09:19.205185 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:19.205129    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2855d516_7749_424b_8bb1_de991fc737cf.slice/crio-b311fffabdcaab44fac9df846863c390cb37e14fb6dd995d032b0362f39fd738 WatchSource:0}: Error finding container b311fffabdcaab44fac9df846863c390cb37e14fb6dd995d032b0362f39fd738: Status 404 returned error can't find the container with id b311fffabdcaab44fac9df846863c390cb37e14fb6dd995d032b0362f39fd738
Oct 31 02:09:19.440899 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:19.440859    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2" event={"ID":"b0f7be2b-866d-4e43-8ee7-b9d711600a64","Type":"ContainerStarted","Data":"242ae2c117cace69267b71dfd971a2797bbf9401093077712ed1406bc77b6b8d"}
Oct 31 02:09:19.441401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:19.440929    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2" event={"ID":"b0f7be2b-866d-4e43-8ee7-b9d711600a64","Type":"ContainerStarted","Data":"f963de6d7dde4640468ee0ab5196af701ce4b2e7e26c350a136b9e56bb6a93f0"}
Oct 31 02:09:19.442446 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:19.442421    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl" event={"ID":"2855d516-7749-424b-8bb1-de991fc737cf","Type":"ContainerStarted","Data":"b311fffabdcaab44fac9df846863c390cb37e14fb6dd995d032b0362f39fd738"}
Oct 31 02:09:19.554757 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:19.554704    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2" podStartSLOduration=1.5546922319999998 podStartE2EDuration="1.554692232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:19.552899836 +0000 UTC m=+3880.853856435" watchObservedRunningTime="2026-10-31 02:09:19.554692232 +0000 UTC m=+3880.855648805"
Oct 31 02:09:20.454389 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:20.454346    5419 generic.go:356] "Generic (PLEG): container finished" podID="2855d516-7749-424b-8bb1-de991fc737cf" containerID="3fc88723f2a2e9bfa0d54ca0b617985bc48d6255a8bf4c9ff8285196656864f8" exitCode=0
Oct 31 02:09:20.454904 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:20.454428    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl" event={"ID":"2855d516-7749-424b-8bb1-de991fc737cf","Type":"ContainerDied","Data":"3fc88723f2a2e9bfa0d54ca0b617985bc48d6255a8bf4c9ff8285196656864f8"}
Oct 31 02:09:20.456532 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:20.456498    5419 generic.go:356] "Generic (PLEG): container finished" podID="b0f7be2b-866d-4e43-8ee7-b9d711600a64" containerID="242ae2c117cace69267b71dfd971a2797bbf9401093077712ed1406bc77b6b8d" exitCode=0
Oct 31 02:09:20.456710 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:20.456579    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2" event={"ID":"b0f7be2b-866d-4e43-8ee7-b9d711600a64","Type":"ContainerDied","Data":"242ae2c117cace69267b71dfd971a2797bbf9401093077712ed1406bc77b6b8d"}
Oct 31 02:09:21.604202 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.604172    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:21.632418 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.632394    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:21.786335 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786304    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca\") pod \"2855d516-7749-424b-8bb1-de991fc737cf\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") "
Oct 31 02:09:21.786505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786391    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca\") pod \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") "
Oct 31 02:09:21.786505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786426    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca\") pod \"2855d516-7749-424b-8bb1-de991fc737cf\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") "
Oct 31 02:09:21.786505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786454    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca\") pod \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") "
Oct 31 02:09:21.786704 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786605    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc\" (UniqueName: \"kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc\") pod \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\" (UID: \"b0f7be2b-866d-4e43-8ee7-b9d711600a64\") "
Oct 31 02:09:21.786704 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786642    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j\" (UniqueName: \"kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j\") pod \"2855d516-7749-424b-8bb1-de991fc737cf\" (UID: \"2855d516-7749-424b-8bb1-de991fc737cf\") "
Oct 31 02:09:21.786819 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786712    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca" pod "2855d516-7749-424b-8bb1-de991fc737cf" (UID: "2855d516-7749-424b-8bb1-de991fc737cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:21.786949 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786925    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca" pod "b0f7be2b-866d-4e43-8ee7-b9d711600a64" (UID: "b0f7be2b-866d-4e43-8ee7-b9d711600a64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:21.787018 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.786991    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:21.788785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.788760    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca" pod "b0f7be2b-866d-4e43-8ee7-b9d711600a64" (UID: "b0f7be2b-866d-4e43-8ee7-b9d711600a64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:21.788904 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.788820    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j" pod "2855d516-7749-424b-8bb1-de991fc737cf" (UID: "2855d516-7749-424b-8bb1-de991fc737cf"). InnerVolumeSpecName "kube-api-access-blj4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:21.789253 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.789223    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc" pod "b0f7be2b-866d-4e43-8ee7-b9d711600a64" (UID: "b0f7be2b-866d-4e43-8ee7-b9d711600a64"). InnerVolumeSpecName "kube-api-access-zx9tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:21.789348 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.789272    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca" pod "2855d516-7749-424b-8bb1-de991fc737cf" (UID: "2855d516-7749-424b-8bb1-de991fc737cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:21.888116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.888082    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:21.888116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.888111    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2855d516-7749-424b-8bb1-de991fc737cf-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:21.888353 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.888123    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f7be2b-866d-4e43-8ee7-b9d711600a64-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:21.888353 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.888133    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx9tc\" (UniqueName: \"kubernetes.io/projected/b0f7be2b-866d-4e43-8ee7-b9d711600a64-kube-api-access-zx9tc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:21.888353 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:21.888164    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blj4j\" (UniqueName: \"kubernetes.io/projected/2855d516-7749-424b-8bb1-de991fc737cf-kube-api-access-blj4j\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:22.470131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:22.470101    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"
Oct 31 02:09:22.470131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:22.470109    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl" event={"ID":"2855d516-7749-424b-8bb1-de991fc737cf","Type":"ContainerDied","Data":"b311fffabdcaab44fac9df846863c390cb37e14fb6dd995d032b0362f39fd738"}
Oct 31 02:09:22.470409 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:22.470180    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b311fffabdcaab44fac9df846863c390cb37e14fb6dd995d032b0362f39fd738"
Oct 31 02:09:22.471617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:22.471585    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2" event={"ID":"b0f7be2b-866d-4e43-8ee7-b9d711600a64","Type":"ContainerDied","Data":"f963de6d7dde4640468ee0ab5196af701ce4b2e7e26c350a136b9e56bb6a93f0"}
Oct 31 02:09:22.471617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:22.471608    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"
Oct 31 02:09:22.471828 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:22.471632    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f963de6d7dde4640468ee0ab5196af701ce4b2e7e26c350a136b9e56bb6a93f0"
Oct 31 02:09:25.553147 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:25.553116    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"]
Oct 31 02:09:25.557944 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:25.557921    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-183266-9hjh2"]
Oct 31 02:09:26.836875 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:26.836847    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b7e1bf-2209-4a47-94cf-af8170430e88" path="/var/lib/kubelet/pods/71b7e1bf-2209-4a47-94cf-af8170430e88/volumes"
Oct 31 02:09:29.885581 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.885548    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"]
Oct 31 02:09:29.885979 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.885966    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b0f7be2b-866d-4e43-8ee7-b9d711600a64" containerName="revoke-keys"
Oct 31 02:09:29.886026 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.885980    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f7be2b-866d-4e43-8ee7-b9d711600a64" containerName="revoke-keys"
Oct 31 02:09:29.886026 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.886010    5419 cpu_manager.go:414] "containerMap: removing container" podUID="2855d516-7749-424b-8bb1-de991fc737cf" containerName="revoke-keys"
Oct 31 02:09:29.886026 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.886015    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2855d516-7749-424b-8bb1-de991fc737cf" containerName="revoke-keys"
Oct 31 02:09:29.886198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.886085    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="2855d516-7749-424b-8bb1-de991fc737cf" containerName="revoke-keys"
Oct 31 02:09:29.886198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.886100    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f7be2b-866d-4e43-8ee7-b9d711600a64" containerName="revoke-keys"
Oct 31 02:09:29.890400 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.890385    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:29.944862 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.944830    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"]
Oct 31 02:09:29.948656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.948637    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:29.964963 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.964944    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:29.965061 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.964992    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:29.965061 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.965019    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlp4s\" (UniqueName: \"kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:29.965177 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.965106    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:29.965215 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.965183    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54dc\" (UniqueName: \"kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:29.965279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.965262    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:29.991647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.991535    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:09:29.991647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.991608    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:09:29.991647 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.991617    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:09:29.991940 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.991667    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:09:29.994456 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:29.994435    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"]
Oct 31 02:09:30.005396 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.005374    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"]
Oct 31 02:09:30.066872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.066837    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.067052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.066933    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.067052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.066988    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.067052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.067013    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlp4s\" (UniqueName: \"kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.067052 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.067048    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.067307 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.067089    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54dc\" (UniqueName: \"kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.067843 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.067785    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.067843 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.067785    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.069317 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.069291    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.069557 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.069537    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.125642 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.125613    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlp4s\" (UniqueName: \"kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s\") pod \"maas-api-revoke-sub-2fcf5072-wbtfm\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.125788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.125730    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54dc\" (UniqueName: \"kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc\") pod \"maas-api-revoke-sub-3ae8608c-s7fzv\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.204006 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.203926    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:30.264283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.262368    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:30.575438 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.575413    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"]
Oct 31 02:09:30.577414 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:30.577387    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d4fa9b_870c_4aeb_b7d5_4a1a1f275955.slice/crio-7f3829a3488e0a37862797e391a40c61e547981073640332d46ec70a1500258b WatchSource:0}: Error finding container 7f3829a3488e0a37862797e391a40c61e547981073640332d46ec70a1500258b: Status 404 returned error can't find the container with id 7f3829a3488e0a37862797e391a40c61e547981073640332d46ec70a1500258b
Oct 31 02:09:30.684944 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:30.684916    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"]
Oct 31 02:09:30.686357 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:30.686331    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54915945_ccd8_4e05_b8f9_256d85f0f112.slice/crio-cc91d762b9e9ace1e658cee726377d9214a1025adc89b1d2013b9e297c897a3e WatchSource:0}: Error finding container cc91d762b9e9ace1e658cee726377d9214a1025adc89b1d2013b9e297c897a3e: Status 404 returned error can't find the container with id cc91d762b9e9ace1e658cee726377d9214a1025adc89b1d2013b9e297c897a3e
Oct 31 02:09:31.532907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.532866    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv" event={"ID":"54915945-ccd8-4e05-b8f9-256d85f0f112","Type":"ContainerStarted","Data":"6697bcd4d2d6293afad235affd45a02742a7d4b411386124dcdebf5bf14ba192"}
Oct 31 02:09:31.533316 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.532950    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv" event={"ID":"54915945-ccd8-4e05-b8f9-256d85f0f112","Type":"ContainerStarted","Data":"cc91d762b9e9ace1e658cee726377d9214a1025adc89b1d2013b9e297c897a3e"}
Oct 31 02:09:31.534609 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.534583    5419 generic.go:356] "Generic (PLEG): container finished" podID="d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" containerID="11894852e8e5702437bff45e3f7ffad5077222bd449c02082110315c69c38f98" exitCode=0
Oct 31 02:09:31.534730 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.534621    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm" event={"ID":"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955","Type":"ContainerDied","Data":"11894852e8e5702437bff45e3f7ffad5077222bd449c02082110315c69c38f98"}
Oct 31 02:09:31.534730 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.534687    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm" event={"ID":"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955","Type":"ContainerStarted","Data":"7f3829a3488e0a37862797e391a40c61e547981073640332d46ec70a1500258b"}
Oct 31 02:09:31.645764 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.645706    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv" podStartSLOduration=2.645691916 podStartE2EDuration="2.645691916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:31.644417252 +0000 UTC m=+3892.945373829" watchObservedRunningTime="2026-10-31 02:09:31.645691916 +0000 UTC m=+3892.946648492"
Oct 31 02:09:31.704496 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.704461    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" containerID="cri-o://b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63" gracePeriod=45
Oct 31 02:09:31.961195 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.961167    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 02:09:31.988506 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988471    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache\") pod \"43590537-8e3d-459c-a696-0222664cf92d\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") "
Oct 31 02:09:31.988634 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988560    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home\") pod \"43590537-8e3d-459c-a696-0222664cf92d\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") "
Oct 31 02:09:31.988634 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988595    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn\" (UniqueName: \"kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn\") pod \"43590537-8e3d-459c-a696-0222664cf92d\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") "
Oct 31 02:09:31.988783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988633    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs\" (UniqueName: \"kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs\") pod \"43590537-8e3d-459c-a696-0222664cf92d\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") "
Oct 31 02:09:31.988783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988680    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm\") pod \"43590537-8e3d-459c-a696-0222664cf92d\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") "
Oct 31 02:09:31.988783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988703    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir\") pod \"43590537-8e3d-459c-a696-0222664cf92d\" (UID: \"43590537-8e3d-459c-a696-0222664cf92d\") "
Oct 31 02:09:31.988783 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988727    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache" pod "43590537-8e3d-459c-a696-0222664cf92d" (UID: "43590537-8e3d-459c-a696-0222664cf92d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:31.988996 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.988841    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home" pod "43590537-8e3d-459c-a696-0222664cf92d" (UID: "43590537-8e3d-459c-a696-0222664cf92d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:31.989058 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.989020    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir" pod "43590537-8e3d-459c-a696-0222664cf92d" (UID: "43590537-8e3d-459c-a696-0222664cf92d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:31.989191 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.989173    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:31.989254 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.989194    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:31.989254 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.989203    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:31.990949 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.990928    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs" pod "43590537-8e3d-459c-a696-0222664cf92d" (UID: "43590537-8e3d-459c-a696-0222664cf92d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:31.991048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.990969    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm" pod "43590537-8e3d-459c-a696-0222664cf92d" (UID: "43590537-8e3d-459c-a696-0222664cf92d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:31.991048 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:31.991013    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn" pod "43590537-8e3d-459c-a696-0222664cf92d" (UID: "43590537-8e3d-459c-a696-0222664cf92d"). InnerVolumeSpecName "kube-api-access-pb5sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:32.090745 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.090717    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb5sn\" (UniqueName: \"kubernetes.io/projected/43590537-8e3d-459c-a696-0222664cf92d-kube-api-access-pb5sn\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.090745 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.090746    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43590537-8e3d-459c-a696-0222664cf92d-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.090917 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.090758    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43590537-8e3d-459c-a696-0222664cf92d-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.542640 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.542604    5419 generic.go:356] "Generic (PLEG): container finished" podID="54915945-ccd8-4e05-b8f9-256d85f0f112" containerID="6697bcd4d2d6293afad235affd45a02742a7d4b411386124dcdebf5bf14ba192" exitCode=0
Oct 31 02:09:32.543123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.542695    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv" event={"ID":"54915945-ccd8-4e05-b8f9-256d85f0f112","Type":"ContainerDied","Data":"6697bcd4d2d6293afad235affd45a02742a7d4b411386124dcdebf5bf14ba192"}
Oct 31 02:09:32.544162 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.544121    5419 generic.go:356] "Generic (PLEG): container finished" podID="43590537-8e3d-459c-a696-0222664cf92d" containerID="b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63" exitCode=0
Oct 31 02:09:32.544266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.544179    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"
Oct 31 02:09:32.544266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.544201    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" event={"ID":"43590537-8e3d-459c-a696-0222664cf92d","Type":"ContainerDied","Data":"b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63"}
Oct 31 02:09:32.544266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.544238    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" event={"ID":"43590537-8e3d-459c-a696-0222664cf92d","Type":"ContainerDied","Data":"1374dd9a31016fbc277a6c49ecadc0c0390f298defa50305fdfd981876c35a75"}
Oct 31 02:09:32.544266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.544256    5419 scope.go:122] "RemoveContainer" containerID="b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63"
Oct 31 02:09:32.556500 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.556471    5419 scope.go:122] "RemoveContainer" containerID="b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63"
Oct 31 02:09:32.556811 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:32.556786    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63\": container with ID starting with b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63 not found: ID does not exist" containerID="b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63"
Oct 31 02:09:32.556920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.556818    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63"} err="failed to get container status \"b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63\": rpc error: code = NotFound desc = could not find container \"b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63\": container with ID starting with b7b8700fc27e38a63fcada36af65e73e455db160a0947058fce80f0d41c1cb63 not found: ID does not exist"
Oct 31 02:09:32.711898 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.711869    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:32.773151 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.773104    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"]
Oct 31 02:09:32.777116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.777087    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm"]
Oct 31 02:09:32.799501 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.799428    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca\") pod \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") "
Oct 31 02:09:32.799501 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.799493    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s\" (UniqueName: \"kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s\") pod \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") "
Oct 31 02:09:32.799716 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.799601    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca\") pod \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\" (UID: \"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955\") "
Oct 31 02:09:32.800039 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.800010    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca" pod "d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" (UID: "d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:32.801699 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.801670    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca" pod "d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" (UID: "d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:32.802155 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.802119    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s" pod "d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" (UID: "d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955"). InnerVolumeSpecName "kube-api-access-vlp4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:32.836541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.836506    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43590537-8e3d-459c-a696-0222664cf92d" path="/var/lib/kubelet/pods/43590537-8e3d-459c-a696-0222664cf92d/volumes"
Oct 31 02:09:32.867197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.867162    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:09:32.900944 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.900910    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.900944 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.900939    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlp4s\" (UniqueName: \"kubernetes.io/projected/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-kube-api-access-vlp4s\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.900944 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.900949    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:32.922055 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:32.922003    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a8e157/simulator-w3-a8e157-kserve-5c554d85ff-8v2gm" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 02:09:33.552514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.552484    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm"
Oct 31 02:09:33.553058 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.552513    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fcf5072-wbtfm" event={"ID":"d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955","Type":"ContainerDied","Data":"7f3829a3488e0a37862797e391a40c61e547981073640332d46ec70a1500258b"}
Oct 31 02:09:33.553058 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.552572    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3829a3488e0a37862797e391a40c61e547981073640332d46ec70a1500258b"
Oct 31 02:09:33.698162 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.698110    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:33.810807 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.810703    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc\" (UniqueName: \"kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc\") pod \"54915945-ccd8-4e05-b8f9-256d85f0f112\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") "
Oct 31 02:09:33.810807 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.810775    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca\") pod \"54915945-ccd8-4e05-b8f9-256d85f0f112\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") "
Oct 31 02:09:33.812038 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.812008    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca\") pod \"54915945-ccd8-4e05-b8f9-256d85f0f112\" (UID: \"54915945-ccd8-4e05-b8f9-256d85f0f112\") "
Oct 31 02:09:33.812747 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.812465    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca" pod "54915945-ccd8-4e05-b8f9-256d85f0f112" (UID: "54915945-ccd8-4e05-b8f9-256d85f0f112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:33.815308 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.815275    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc" pod "54915945-ccd8-4e05-b8f9-256d85f0f112" (UID: "54915945-ccd8-4e05-b8f9-256d85f0f112"). InnerVolumeSpecName "kube-api-access-c54dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:33.815510 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.815492    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca" pod "54915945-ccd8-4e05-b8f9-256d85f0f112" (UID: "54915945-ccd8-4e05-b8f9-256d85f0f112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:33.913266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.913230    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:33.913266 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.913267    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54915945-ccd8-4e05-b8f9-256d85f0f112-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:33.913488 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:33.913282    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54dc\" (UniqueName: \"kubernetes.io/projected/54915945-ccd8-4e05-b8f9-256d85f0f112-kube-api-access-c54dc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:34.562281 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:34.562242    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv" event={"ID":"54915945-ccd8-4e05-b8f9-256d85f0f112","Type":"ContainerDied","Data":"cc91d762b9e9ace1e658cee726377d9214a1025adc89b1d2013b9e297c897a3e"}
Oct 31 02:09:34.562281 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:34.562275    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae8608c-s7fzv"
Oct 31 02:09:34.562798 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:34.562305    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc91d762b9e9ace1e658cee726377d9214a1025adc89b1d2013b9e297c897a3e"
Oct 31 02:09:37.442881 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:37.442846    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"]
Oct 31 02:09:37.492119 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:37.492084    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-9e3b95-vpj4d"]
Oct 31 02:09:38.836678 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:38.836646    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ce8675-d4c6-4cd6-ad43-26db7c37bc3a" path="/var/lib/kubelet/pods/10ce8675-d4c6-4cd6-ad43-26db7c37bc3a/volumes"
Oct 31 02:09:41.960311 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:41.960272    5419 scope.go:122] "RemoveContainer" containerID="70fdf40fdb7e7b63e1e54ec03c5c8c880349171ae0715fd4869736b6f92aff5e"
Oct 31 02:09:43.024287 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024252    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"]
Oct 31 02:09:43.024700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024687    5419 cpu_manager.go:414] "containerMap: removing container" podUID="54915945-ccd8-4e05-b8f9-256d85f0f112" containerName="revoke-keys"
Oct 31 02:09:43.024700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024700    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54915945-ccd8-4e05-b8f9-256d85f0f112" containerName="revoke-keys"
Oct 31 02:09:43.024786 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024723    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" containerName="revoke-keys"
Oct 31 02:09:43.024786 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024728    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" containerName="revoke-keys"
Oct 31 02:09:43.024786 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024746    5419 cpu_manager.go:414] "containerMap: removing container" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main"
Oct 31 02:09:43.024786 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024751    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main"
Oct 31 02:09:43.024920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024808    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="43590537-8e3d-459c-a696-0222664cf92d" containerName="main"
Oct 31 02:09:43.024920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024819    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="54915945-ccd8-4e05-b8f9-256d85f0f112" containerName="revoke-keys"
Oct 31 02:09:43.024920 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.024828    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955" containerName="revoke-keys"
Oct 31 02:09:43.027900 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.027884    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.090374 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.090312    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:09:43.090726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.090393    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:09:43.105240 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.105200    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.105398 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.105366    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.105476 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.105436    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8j4c\" (UniqueName: \"kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.122170 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.122107    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:09:43.122595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.122571    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:09:43.124960 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.124932    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"]
Oct 31 02:09:43.206149 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.206102    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8j4c\" (UniqueName: \"kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.206367 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.206159    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.206367 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.206244    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.206963 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.206934    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.211719 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.211689    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.265791 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.265755    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8j4c\" (UniqueName: \"kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.341304 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.341212    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:43.712084 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:43.712058    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"]
Oct 31 02:09:43.714213 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:43.714179    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d973dc_e4c4_4424_83be_b7cd73a5ac0c.slice/crio-df6b477b66d5db2f60dd64cf5507066344848ec68c457f045fa8e345ad9b6822 WatchSource:0}: Error finding container df6b477b66d5db2f60dd64cf5507066344848ec68c457f045fa8e345ad9b6822: Status 404 returned error can't find the container with id df6b477b66d5db2f60dd64cf5507066344848ec68c457f045fa8e345ad9b6822
Oct 31 02:09:44.583012 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:44.582980    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:09:44.637077 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:44.637045    5419 generic.go:356] "Generic (PLEG): container finished" podID="75d973dc-e4c4-4424-83be-b7cd73a5ac0c" containerID="4b9e4026b2b7eb794a5edfe90fd23c4fde9024fc8b5059b24f76d659c00766f6" exitCode=0
Oct 31 02:09:44.637287 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:44.637150    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz" event={"ID":"75d973dc-e4c4-4424-83be-b7cd73a5ac0c","Type":"ContainerDied","Data":"4b9e4026b2b7eb794a5edfe90fd23c4fde9024fc8b5059b24f76d659c00766f6"}
Oct 31 02:09:44.637287 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:44.637215    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz" event={"ID":"75d973dc-e4c4-4424-83be-b7cd73a5ac0c","Type":"ContainerStarted","Data":"df6b477b66d5db2f60dd64cf5507066344848ec68c457f045fa8e345ad9b6822"}
Oct 31 02:09:44.804732 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:44.804654    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"]
Oct 31 02:09:44.804910 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:44.804873    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" containerID="cri-o://78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3" gracePeriod=30
Oct 31 02:09:45.677377 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.677341    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"]
Oct 31 02:09:45.681008 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.680987    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.739595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.739558    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"]
Oct 31 02:09:45.742863 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.742830    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6nq\" (UniqueName: \"kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.743015 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.742883    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.743173 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.743131    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.780474 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.780450    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:45.843858 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.843832    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca\") pod \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") "
Oct 31 02:09:45.844038 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.843968    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c\" (UniqueName: \"kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c\") pod \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") "
Oct 31 02:09:45.844038 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.844005    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca\") pod \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\" (UID: \"75d973dc-e4c4-4424-83be-b7cd73a5ac0c\") "
Oct 31 02:09:45.844211 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.844092    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6nq\" (UniqueName: \"kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.844211 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.844117    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.844334 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.844270    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.844541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.844511    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca" pod "75d973dc-e4c4-4424-83be-b7cd73a5ac0c" (UID: "75d973dc-e4c4-4424-83be-b7cd73a5ac0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:45.844931 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.844913    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.846624 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.846607    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.846767 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.846750    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca" pod "75d973dc-e4c4-4424-83be-b7cd73a5ac0c" (UID: "75d973dc-e4c4-4424-83be-b7cd73a5ac0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:45.846825 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.846778    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c" pod "75d973dc-e4c4-4424-83be-b7cd73a5ac0c" (UID: "75d973dc-e4c4-4424-83be-b7cd73a5ac0c"). InnerVolumeSpecName "kube-api-access-j8j4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:45.902224 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.900699    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6nq\" (UniqueName: \"kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:45.946096 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.946060    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8j4c\" (UniqueName: \"kubernetes.io/projected/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-kube-api-access-j8j4c\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:45.946096 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.946091    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:45.946366 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.946106    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d973dc-e4c4-4424-83be-b7cd73a5ac0c-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:45.994552 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:45.994513    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:46.373791 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:46.373766    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"]
Oct 31 02:09:46.375661 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:09:46.375631    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a04cc1_7fb2_4423_9cbe_6069b30b2e4d.slice/crio-34cf0a26fd86daf19c10f5fe432084e549ddc0cd4f680fb9098cb1d278b81cae WatchSource:0}: Error finding container 34cf0a26fd86daf19c10f5fe432084e549ddc0cd4f680fb9098cb1d278b81cae: Status 404 returned error can't find the container with id 34cf0a26fd86daf19c10f5fe432084e549ddc0cd4f680fb9098cb1d278b81cae
Oct 31 02:09:46.650893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:46.650804    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"
Oct 31 02:09:46.650893 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:46.650847    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz" event={"ID":"75d973dc-e4c4-4424-83be-b7cd73a5ac0c","Type":"ContainerDied","Data":"df6b477b66d5db2f60dd64cf5507066344848ec68c457f045fa8e345ad9b6822"}
Oct 31 02:09:46.651391 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:46.650904    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6b477b66d5db2f60dd64cf5507066344848ec68c457f045fa8e345ad9b6822"
Oct 31 02:09:46.652360 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:46.652338    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq" event={"ID":"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d","Type":"ContainerStarted","Data":"34cf0a26fd86daf19c10f5fe432084e549ddc0cd4f680fb9098cb1d278b81cae"}
Oct 31 02:09:47.660535 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:47.660448    5419 generic.go:356] "Generic (PLEG): container finished" podID="a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" containerID="56a4894c76011bf6e76d35a48407deaaa96e409a7b2ba82fd37122d3bd1cb63a" exitCode=0
Oct 31 02:09:47.660897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:47.660534    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq" event={"ID":"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d","Type":"ContainerDied","Data":"56a4894c76011bf6e76d35a48407deaaa96e409a7b2ba82fd37122d3bd1cb63a"}
Oct 31 02:09:48.265921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.265884    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:09:48.266109 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.265934    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:09:48.807738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.807712    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:48.876158 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.876101    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca\") pod \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") "
Oct 31 02:09:48.876338 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.876261    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq\" (UniqueName: \"kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq\") pod \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") "
Oct 31 02:09:48.876338 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.876303    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca\") pod \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\" (UID: \"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d\") "
Oct 31 02:09:48.876701 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.876679    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca" pod "a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" (UID: "a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:48.878359 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.878340    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca" pod "a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" (UID: "a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:09:48.878441 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.878414    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq" pod "a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" (UID: "a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d"). InnerVolumeSpecName "kube-api-access-sq6nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:48.978235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.978193    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6nq\" (UniqueName: \"kubernetes.io/projected/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-kube-api-access-sq6nq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:48.978235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.978224    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:48.978454 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:48.978240    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:49.674326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:49.674290    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq" event={"ID":"a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d","Type":"ContainerDied","Data":"34cf0a26fd86daf19c10f5fe432084e549ddc0cd4f680fb9098cb1d278b81cae"}
Oct 31 02:09:49.674326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:49.674311    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"
Oct 31 02:09:49.674540 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:49.674339    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34cf0a26fd86daf19c10f5fe432084e549ddc0cd4f680fb9098cb1d278b81cae"
Oct 31 02:09:50.052164 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.052126    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 02:09:50.190834 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.190804    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.190843    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.190882    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.190960    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191045 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.190993    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191054    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191092    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191166    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191199    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:50.191313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191211    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data\") pod \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\" (UID: \"38dc283f-bbcb-4544-8f81-13a0d10c52f6\") "
Oct 31 02:09:50.191530 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191383    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:50.191530 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191448    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:50.191530 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191403    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:50.191689 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191595    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:50.191755 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191732    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.191826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191760    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.191826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191775    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.191826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191788    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.191826 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.191800    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.193837 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.193806    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:09:50.193954 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.193845    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:09:50.194039 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.194020    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:50.195297 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.195255    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9" pod "38dc283f-bbcb-4544-8f81-13a0d10c52f6" (UID: "38dc283f-bbcb-4544-8f81-13a0d10c52f6"). InnerVolumeSpecName "kube-api-access-mltq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:50.292514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.292430    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.292514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.292458    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mltq9\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-kube-api-access-mltq9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.292514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.292468    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.292514 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.292478    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc283f-bbcb-4544-8f81-13a0d10c52f6-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:50.682550 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.682452    5419 generic.go:356] "Generic (PLEG): container finished" podID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerID="78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3" exitCode=0
Oct 31 02:09:50.682550 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.682510    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"
Oct 31 02:09:50.682550 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.682541    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" event={"ID":"38dc283f-bbcb-4544-8f81-13a0d10c52f6","Type":"ContainerDied","Data":"78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3"}
Oct 31 02:09:50.682850 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.682594    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl" event={"ID":"38dc283f-bbcb-4544-8f81-13a0d10c52f6","Type":"ContainerDied","Data":"bac81672f321151aa9805b5b7b330cb6bb6abce82a19fce0a3d4c3b037c86357"}
Oct 31 02:09:50.682850 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.682616    5419 scope.go:122] "RemoveContainer" containerID="78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3"
Oct 31 02:09:50.692963 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.692942    5419 scope.go:122] "RemoveContainer" containerID="6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"
Oct 31 02:09:50.702094 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.702077    5419 scope.go:122] "RemoveContainer" containerID="78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3"
Oct 31 02:09:50.702412 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:50.702391    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3\": container with ID starting with 78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3 not found: ID does not exist" containerID="78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3"
Oct 31 02:09:50.702582 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.702420    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3"} err="failed to get container status \"78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3\": rpc error: code = NotFound desc = could not find container \"78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3\": container with ID starting with 78e03240939e6e2561508a6bb6272d587adf4404f2a2c240cfd3c6559fa0bbc3 not found: ID does not exist"
Oct 31 02:09:50.702582 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.702447    5419 scope.go:122] "RemoveContainer" containerID="6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"
Oct 31 02:09:50.702715 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:09:50.702696    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579\": container with ID starting with 6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579 not found: ID does not exist" containerID="6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"
Oct 31 02:09:50.702755 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.702723    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579"} err="failed to get container status \"6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579\": rpc error: code = NotFound desc = could not find container \"6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579\": container with ID starting with 6ef2db8909f55cc5a4ef3683d2329af5f79152313cc8057965958f971e181579 not found: ID does not exist"
Oct 31 02:09:50.844777 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.844743    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"]
Oct 31 02:09:50.894285 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:50.894255    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9e3b95-openshift-default-6cdb7fb797-kr8vl"]
Oct 31 02:09:52.836364 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:52.836335    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" path="/var/lib/kubelet/pods/38dc283f-bbcb-4544-8f81-13a0d10c52f6/volumes"
Oct 31 02:09:54.548795 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:54.548755    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"]
Oct 31 02:09:54.654354 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:54.654318    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"]
Oct 31 02:09:56.999552 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:56.999508    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"]
Oct 31 02:09:58.652700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:58.652668    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"]
Oct 31 02:09:58.700115 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:58.700079    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057e2d-bz79x"]
Oct 31 02:09:58.836069 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:58.836038    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0175d9a-1e2b-427d-adfb-335128abd607" path="/var/lib/kubelet/pods/d0175d9a-1e2b-427d-adfb-335128abd607/volumes"
Oct 31 02:09:59.549588 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.549533    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx" podUID="ae7c7220-c681-4c53-b18e-05ff582dcf0d" containerName="payload-processing" containerID="cri-o://b5c699ea8902e5ad26d943dac0f4c920bf83d45d9086c38b80f6eba186665855" gracePeriod=25
Oct 31 02:09:59.655573 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.655487    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw" podUID="f32ee054-d11f-4913-bc14-3cd904576f22" containerName="payload-pre-processing" containerID="cri-o://34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969" gracePeriod=25
Oct 31 02:09:59.743994 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.743950    5419 generic.go:356] "Generic (PLEG): container finished" podID="ae7c7220-c681-4c53-b18e-05ff582dcf0d" containerID="b5c699ea8902e5ad26d943dac0f4c920bf83d45d9086c38b80f6eba186665855" exitCode=0
Oct 31 02:09:59.744210 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.744094    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx" event={"ID":"ae7c7220-c681-4c53-b18e-05ff582dcf0d","Type":"ContainerDied","Data":"b5c699ea8902e5ad26d943dac0f4c920bf83d45d9086c38b80f6eba186665855"}
Oct 31 02:09:59.794891 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.794857    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 02:09:59.896802 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.896770    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume\") pod \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") "
Oct 31 02:09:59.896987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.896821    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n\" (UniqueName: \"kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n\") pod \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\" (UID: \"ae7c7220-c681-4c53-b18e-05ff582dcf0d\") "
Oct 31 02:09:59.897174 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.897148    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume" pod "ae7c7220-c681-4c53-b18e-05ff582dcf0d" (UID: "ae7c7220-c681-4c53-b18e-05ff582dcf0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:09:59.899182 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.899161    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n" pod "ae7c7220-c681-4c53-b18e-05ff582dcf0d" (UID: "ae7c7220-c681-4c53-b18e-05ff582dcf0d"). InnerVolumeSpecName "kube-api-access-srv2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:09:59.931495 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.931467    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 02:09:59.998118 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.998087    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srv2n\" (UniqueName: \"kubernetes.io/projected/ae7c7220-c681-4c53-b18e-05ff582dcf0d-kube-api-access-srv2n\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:09:59.998118 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:09:59.998112    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae7c7220-c681-4c53-b18e-05ff582dcf0d-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:00.098781 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.098700    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume\") pod \"f32ee054-d11f-4913-bc14-3cd904576f22\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") "
Oct 31 02:10:00.098781 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.098777    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz\" (UniqueName: \"kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz\") pod \"f32ee054-d11f-4913-bc14-3cd904576f22\" (UID: \"f32ee054-d11f-4913-bc14-3cd904576f22\") "
Oct 31 02:10:00.099133 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.099110    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume" pod "f32ee054-d11f-4913-bc14-3cd904576f22" (UID: "f32ee054-d11f-4913-bc14-3cd904576f22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:00.101011 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.100988    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz" pod "f32ee054-d11f-4913-bc14-3cd904576f22" (UID: "f32ee054-d11f-4913-bc14-3cd904576f22"). InnerVolumeSpecName "kube-api-access-ch6nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:00.200224 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.200198    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32ee054-d11f-4913-bc14-3cd904576f22-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:00.200224 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.200223    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6nz\" (UniqueName: \"kubernetes.io/projected/f32ee054-d11f-4913-bc14-3cd904576f22-kube-api-access-ch6nz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:00.751221 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.751187    5419 generic.go:356] "Generic (PLEG): container finished" podID="f32ee054-d11f-4913-bc14-3cd904576f22" containerID="34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969" exitCode=0
Oct 31 02:10:00.751737 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.751235    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"
Oct 31 02:10:00.751737 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.751270    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw" event={"ID":"f32ee054-d11f-4913-bc14-3cd904576f22","Type":"ContainerDied","Data":"34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969"}
Oct 31 02:10:00.751737 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.751327    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw" event={"ID":"f32ee054-d11f-4913-bc14-3cd904576f22","Type":"ContainerDied","Data":"a0afd4089d481d07f5902f9fa5de3bcfff84646357246b7becc6c40f866e40e9"}
Oct 31 02:10:00.751737 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.751356    5419 scope.go:122] "RemoveContainer" containerID="34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969"
Oct 31 02:10:00.753013 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.752993    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx" event={"ID":"ae7c7220-c681-4c53-b18e-05ff582dcf0d","Type":"ContainerDied","Data":"22fd5dc050b4985035dcb7dabbfd4ab226de1d73ddff40c181f089f56b00138d"}
Oct 31 02:10:00.753175 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.753002    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"
Oct 31 02:10:00.761759 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.761739    5419 scope.go:122] "RemoveContainer" containerID="34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969"
Oct 31 02:10:00.762118 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:00.762094    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969\": container with ID starting with 34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969 not found: ID does not exist" containerID="34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969"
Oct 31 02:10:00.762236 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.762128    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969"} err="failed to get container status \"34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969\": rpc error: code = NotFound desc = could not find container \"34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969\": container with ID starting with 34666aad22978dc643e75b6647edb78ec2c60aeb401c5e5a446b9df05f29f969 not found: ID does not exist"
Oct 31 02:10:00.762236 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.762191    5419 scope.go:122] "RemoveContainer" containerID="b5c699ea8902e5ad26d943dac0f4c920bf83d45d9086c38b80f6eba186665855"
Oct 31 02:10:00.912861 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.912826    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"]
Oct 31 02:10:00.962062 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:00.962028    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5e916d-df7b4777f-92nmw"]
Oct 31 02:10:01.069359 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:01.069268    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"]
Oct 31 02:10:01.073116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:01.073091    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5e916d-b7f54ff5f-hcptx"]
Oct 31 02:10:02.000606 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.000567    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2" podUID="db617360-e93d-4e60-be41-83bf1e6f8591" containerName="payload-processing" containerID="cri-o://80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15" gracePeriod=25
Oct 31 02:10:02.248592 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.248564    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 02:10:02.421356 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.421285    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume\") pod \"db617360-e93d-4e60-be41-83bf1e6f8591\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") "
Oct 31 02:10:02.421356 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.421319    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s\" (UniqueName: \"kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s\") pod \"db617360-e93d-4e60-be41-83bf1e6f8591\" (UID: \"db617360-e93d-4e60-be41-83bf1e6f8591\") "
Oct 31 02:10:02.421658 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.421636    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume" pod "db617360-e93d-4e60-be41-83bf1e6f8591" (UID: "db617360-e93d-4e60-be41-83bf1e6f8591"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:02.423585 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.423558    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s" pod "db617360-e93d-4e60-be41-83bf1e6f8591" (UID: "db617360-e93d-4e60-be41-83bf1e6f8591"). InnerVolumeSpecName "kube-api-access-kmb2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:02.522743 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.522711    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db617360-e93d-4e60-be41-83bf1e6f8591-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:02.522743 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.522740    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmb2s\" (UniqueName: \"kubernetes.io/projected/db617360-e93d-4e60-be41-83bf1e6f8591-kube-api-access-kmb2s\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:02.768449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.768415    5419 generic.go:356] "Generic (PLEG): container finished" podID="db617360-e93d-4e60-be41-83bf1e6f8591" containerID="80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15" exitCode=0
Oct 31 02:10:02.768656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.768455    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"
Oct 31 02:10:02.768656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.768459    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2" event={"ID":"db617360-e93d-4e60-be41-83bf1e6f8591","Type":"ContainerDied","Data":"80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15"}
Oct 31 02:10:02.768656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.768634    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2" event={"ID":"db617360-e93d-4e60-be41-83bf1e6f8591","Type":"ContainerDied","Data":"a514cc65c99bf1ca94a3635232333b6ab5ea9f50ebd94957ef568e28e02f3a33"}
Oct 31 02:10:02.768821 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.768668    5419 scope.go:122] "RemoveContainer" containerID="80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15"
Oct 31 02:10:02.779210 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.779191    5419 scope.go:122] "RemoveContainer" containerID="80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15"
Oct 31 02:10:02.779519 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:02.779500    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15\": container with ID starting with 80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15 not found: ID does not exist" containerID="80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15"
Oct 31 02:10:02.779605 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.779531    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15"} err="failed to get container status \"80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15\": rpc error: code = NotFound desc = could not find container \"80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15\": container with ID starting with 80cde101b0333cb6d10dcbcc176251fda4df9e84a50d5bf0a5d0e402faf29a15 not found: ID does not exist"
Oct 31 02:10:02.839908 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.839877    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7c7220-c681-4c53-b18e-05ff582dcf0d" path="/var/lib/kubelet/pods/ae7c7220-c681-4c53-b18e-05ff582dcf0d/volumes"
Oct 31 02:10:02.840289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.840270    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32ee054-d11f-4913-bc14-3cd904576f22" path="/var/lib/kubelet/pods/f32ee054-d11f-4913-bc14-3cd904576f22/volumes"
Oct 31 02:10:02.932288 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.932252    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"]
Oct 31 02:10:02.938564 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:02.938538    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5632ea-8694dd957f-hrfm2"]
Oct 31 02:10:04.836003 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:04.835975    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db617360-e93d-4e60-be41-83bf1e6f8591" path="/var/lib/kubelet/pods/db617360-e93d-4e60-be41-83bf1e6f8591/volumes"
Oct 31 02:10:05.085435 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:05.085403    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"]
Oct 31 02:10:05.134736 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:05.134658    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-a8e157-r4dls"]
Oct 31 02:10:06.374208 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:06.374174    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"]
Oct 31 02:10:06.423037 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:06.423000    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5e916d-qhwnz"]
Oct 31 02:10:06.836507 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:06.836476    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d008513-4ca3-479e-bcf2-6a7d8cb0d932" path="/var/lib/kubelet/pods/6d008513-4ca3-479e-bcf2-6a7d8cb0d932/volumes"
Oct 31 02:10:06.836882 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:06.836870    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d973dc-e4c4-4424-83be-b7cd73a5ac0c" path="/var/lib/kubelet/pods/75d973dc-e4c4-4424-83be-b7cd73a5ac0c/volumes"
Oct 31 02:10:09.692975 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:09.692938    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:10:12.559733 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:12.559701    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"]
Oct 31 02:10:12.608288 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:12.608250    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5632ea-lzxfq"]
Oct 31 02:10:12.836938 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:12.836859    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" path="/var/lib/kubelet/pods/a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d/volumes"
Oct 31 02:10:14.581530 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:14.581485    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:10:19.358209 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:19.358172    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"]
Oct 31 02:10:19.358605 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:19.358393    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy" containerID="cri-o://7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e" gracePeriod=30
Oct 31 02:10:20.317344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317309    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"]
Oct 31 02:10:20.317738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317725    5419 cpu_manager.go:414] "containerMap: removing container" podUID="a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" containerName="revoke-keys"
Oct 31 02:10:20.317785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317741    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" containerName="revoke-keys"
Oct 31 02:10:20.317785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317749    5419 cpu_manager.go:414] "containerMap: removing container" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy"
Oct 31 02:10:20.317785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317754    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy"
Oct 31 02:10:20.317785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317771    5419 cpu_manager.go:414] "containerMap: removing container" podUID="75d973dc-e4c4-4424-83be-b7cd73a5ac0c" containerName="revoke-keys"
Oct 31 02:10:20.317785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317777    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d973dc-e4c4-4424-83be-b7cd73a5ac0c" containerName="revoke-keys"
Oct 31 02:10:20.317785 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317784    5419 cpu_manager.go:414] "containerMap: removing container" podUID="ae7c7220-c681-4c53-b18e-05ff582dcf0d" containerName="payload-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317788    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7c7220-c681-4c53-b18e-05ff582dcf0d" containerName="payload-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317799    5419 cpu_manager.go:414] "containerMap: removing container" podUID="f32ee054-d11f-4913-bc14-3cd904576f22" containerName="payload-pre-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317803    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32ee054-d11f-4913-bc14-3cd904576f22" containerName="payload-pre-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317811    5419 cpu_manager.go:414] "containerMap: removing container" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317815    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317821    5419 cpu_manager.go:414] "containerMap: removing container" podUID="db617360-e93d-4e60-be41-83bf1e6f8591" containerName="payload-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317826    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db617360-e93d-4e60-be41-83bf1e6f8591" containerName="payload-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317880    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32ee054-d11f-4913-bc14-3cd904576f22" containerName="payload-pre-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317887    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317893    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="db617360-e93d-4e60-be41-83bf1e6f8591" containerName="payload-processing"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317904    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a04cc1-7fb2-4423-9cbe-6069b30b2e4d" containerName="revoke-keys"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317912    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d973dc-e4c4-4424-83be-b7cd73a5ac0c" containerName="revoke-keys"
Oct 31 02:10:20.317980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.317918    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7c7220-c681-4c53-b18e-05ff582dcf0d" containerName="payload-processing"
Oct 31 02:10:20.321325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.321308    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.378349 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.378316    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"]
Oct 31 02:10:20.405977 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.405944    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5klv\" (UniqueName: \"kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.406184 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.405986    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.406184 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.406076    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.424869 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.424830    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:10:20.424869 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.424836    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:10:20.424869 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.424857    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:10:20.425072 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.424934    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:10:20.507688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.507646    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5klv\" (UniqueName: \"kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.507885 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.507700    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.507885 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.507731    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.508544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.508465    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.510388 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.510360    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.564126 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.564093    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5klv\" (UniqueName: \"kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv\") pod \"maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:20.634329 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:20.634259    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:21.007165 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:21.007121    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"]
Oct 31 02:10:21.009574 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:10:21.009546    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa354fec_85b8_45d1_b6f6_40499a06317a.slice/crio-dab4dca218326576b579d48d43ae7ae932a36587bd72191cb733173aecd94b1e WatchSource:0}: Error finding container dab4dca218326576b579d48d43ae7ae932a36587bd72191cb733173aecd94b1e: Status 404 returned error can't find the container with id dab4dca218326576b579d48d43ae7ae932a36587bd72191cb733173aecd94b1e
Oct 31 02:10:21.046764 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:21.046740    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"]
Oct 31 02:10:21.048717 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:21.048692    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee2c2b5-tdhfn"]
Oct 31 02:10:21.895525 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:21.895490    5419 generic.go:356] "Generic (PLEG): container finished" podID="aa354fec-85b8-45d1-b6f6-40499a06317a" containerID="4fb2091d2bf6bd023ae06fd79e2feb390d02559670802b6dea05dced7d7f993c" exitCode=0
Oct 31 02:10:21.895892 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:21.895567    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h" event={"ID":"aa354fec-85b8-45d1-b6f6-40499a06317a","Type":"ContainerDied","Data":"4fb2091d2bf6bd023ae06fd79e2feb390d02559670802b6dea05dced7d7f993c"}
Oct 31 02:10:21.895892 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:21.895625    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h" event={"ID":"aa354fec-85b8-45d1-b6f6-40499a06317a","Type":"ContainerStarted","Data":"dab4dca218326576b579d48d43ae7ae932a36587bd72191cb733173aecd94b1e"}
Oct 31 02:10:22.837497 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:22.837469    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c3fd9e-0272-4714-a17b-8f3556235ef5" path="/var/lib/kubelet/pods/11c3fd9e-0272-4714-a17b-8f3556235ef5/volumes"
Oct 31 02:10:22.976051 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:22.976020    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"]
Oct 31 02:10:22.976486 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:22.976305    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" containerID="cri-o://6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22" gracePeriod=30
Oct 31 02:10:23.061241 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.061082    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:23.129897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.129814    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca\") pod \"aa354fec-85b8-45d1-b6f6-40499a06317a\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") "
Oct 31 02:10:23.129897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.129868    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca\") pod \"aa354fec-85b8-45d1-b6f6-40499a06317a\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") "
Oct 31 02:10:23.130090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.129968    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv\" (UniqueName: \"kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv\") pod \"aa354fec-85b8-45d1-b6f6-40499a06317a\" (UID: \"aa354fec-85b8-45d1-b6f6-40499a06317a\") "
Oct 31 02:10:23.130250 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.130223    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca" pod "aa354fec-85b8-45d1-b6f6-40499a06317a" (UID: "aa354fec-85b8-45d1-b6f6-40499a06317a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:23.132175 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.132131    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv" pod "aa354fec-85b8-45d1-b6f6-40499a06317a" (UID: "aa354fec-85b8-45d1-b6f6-40499a06317a"). InnerVolumeSpecName "kube-api-access-f5klv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:23.132304 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.132188    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca" pod "aa354fec-85b8-45d1-b6f6-40499a06317a" (UID: "aa354fec-85b8-45d1-b6f6-40499a06317a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:23.231325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.231295    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:23.231325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.231324    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa354fec-85b8-45d1-b6f6-40499a06317a-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:23.231527 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.231336    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5klv\" (UniqueName: \"kubernetes.io/projected/aa354fec-85b8-45d1-b6f6-40499a06317a-kube-api-access-f5klv\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:23.790711 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.790670    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"]
Oct 31 02:10:23.791309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.791288    5419 cpu_manager.go:414] "containerMap: removing container" podUID="aa354fec-85b8-45d1-b6f6-40499a06317a" containerName="revoke-keys"
Oct 31 02:10:23.791309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.791308    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa354fec-85b8-45d1-b6f6-40499a06317a" containerName="revoke-keys"
Oct 31 02:10:23.791488 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.791389    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa354fec-85b8-45d1-b6f6-40499a06317a" containerName="revoke-keys"
Oct 31 02:10:23.791488 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.791404    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="38dc283f-bbcb-4544-8f81-13a0d10c52f6" containerName="istio-proxy"
Oct 31 02:10:23.794591 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.794576    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.837195 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.837169    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqpdj\" (UniqueName: \"kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.837359 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.837300    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.837403 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.837360    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.850937 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.850908    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"]
Oct 31 02:10:23.908858 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.908787    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h" event={"ID":"aa354fec-85b8-45d1-b6f6-40499a06317a","Type":"ContainerDied","Data":"dab4dca218326576b579d48d43ae7ae932a36587bd72191cb733173aecd94b1e"}
Oct 31 02:10:23.909187 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.909159    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dab4dca218326576b579d48d43ae7ae932a36587bd72191cb733173aecd94b1e"
Oct 31 02:10:23.909187 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.908882    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"
Oct 31 02:10:23.938788 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.938761    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.938930 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.938915    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqpdj\" (UniqueName: \"kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.939587 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.939383    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.940337 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.940315    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.941574 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.941550    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:23.994936 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:23.994897    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqpdj\" (UniqueName: \"kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:24.111462 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.111385    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:24.485171 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:10:24.484695    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eecb47e_7a7c_4fc7_8b21_f016b524912e.slice/crio-0d9db81384e7a61ea137e74ab41ee415818713669d69b198799608e5d78014e7 WatchSource:0}: Error finding container 0d9db81384e7a61ea137e74ab41ee415818713669d69b198799608e5d78014e7: Status 404 returned error can't find the container with id 0d9db81384e7a61ea137e74ab41ee415818713669d69b198799608e5d78014e7
Oct 31 02:10:24.485171 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.484880    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"]
Oct 31 02:10:24.613252 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.613227    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 02:10:24.747828 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.747788    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.747835    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.747864    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.747889    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.747941    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748021 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748004    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748092    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748125    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748181    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token\") pod \"606c61cc-b66d-44a3-adca-c194a4879ea9\" (UID: \"606c61cc-b66d-44a3-adca-c194a4879ea9\") "
Oct 31 02:10:24.748566 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748172    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:24.748645 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748257    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:24.748645 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748285    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:24.748645 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748487    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:24.748645 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.748589    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:24.750551 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.750528    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:24.750656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.750595    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "kube-api-access-x5blz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:24.750928 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.750906    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:24.751050 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.750941    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token" pod "606c61cc-b66d-44a3-adca-c194a4879ea9" (UID: "606c61cc-b66d-44a3-adca-c194a4879ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:24.798635 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.798606    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"]
Oct 31 02:10:24.818540 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.818510    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"]
Oct 31 02:10:24.849469 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849442    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849469 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849470    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c61cc-b66d-44a3-adca-c194a4879ea9-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849483    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5blz\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-kube-api-access-x5blz\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849495    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849509    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849521    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849532    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849544    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.849653 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.849556    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c61cc-b66d-44a3-adca-c194a4879ea9-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:24.917720 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.917690    5419 generic.go:356] "Generic (PLEG): container finished" podID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerID="7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e" exitCode=0
Oct 31 02:10:24.917905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.917777    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" event={"ID":"606c61cc-b66d-44a3-adca-c194a4879ea9","Type":"ContainerDied","Data":"7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e"}
Oct 31 02:10:24.917905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.917846    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc" event={"ID":"606c61cc-b66d-44a3-adca-c194a4879ea9","Type":"ContainerDied","Data":"b2725f6a22db1b6bbb1f22c32c7fc99a581a8506f3c8b9ef495dc71719977b0d"}
Oct 31 02:10:24.917905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.917788    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"
Oct 31 02:10:24.917905 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.917877    5419 scope.go:122] "RemoveContainer" containerID="7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e"
Oct 31 02:10:24.920260 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.920222    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt" event={"ID":"7eecb47e-7a7c-4fc7-8b21-f016b524912e","Type":"ContainerStarted","Data":"2ab2c44ee9aa53419064d6a4dfdbc699da901199d4b3429aa454252fce56e2f0"}
Oct 31 02:10:24.920576 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.920526    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt" event={"ID":"7eecb47e-7a7c-4fc7-8b21-f016b524912e","Type":"ContainerStarted","Data":"0d9db81384e7a61ea137e74ab41ee415818713669d69b198799608e5d78014e7"}
Oct 31 02:10:24.930695 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.930672    5419 scope.go:122] "RemoveContainer" containerID="72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"
Oct 31 02:10:24.941529 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.941512    5419 scope.go:122] "RemoveContainer" containerID="7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e"
Oct 31 02:10:24.941808 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:24.941789    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e\": container with ID starting with 7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e not found: ID does not exist" containerID="7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e"
Oct 31 02:10:24.941863 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.941818    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e"} err="failed to get container status \"7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e\": rpc error: code = NotFound desc = could not find container \"7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e\": container with ID starting with 7a7fcd11d0dd2be231eb0d37a7b5d08976b2cd2c824a26bbfe5624d238cd912e not found: ID does not exist"
Oct 31 02:10:24.941863 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.941835    5419 scope.go:122] "RemoveContainer" containerID="72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"
Oct 31 02:10:24.942070 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:24.942052    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c\": container with ID starting with 72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c not found: ID does not exist" containerID="72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"
Oct 31 02:10:24.942116 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:24.942075    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c"} err="failed to get container status \"72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c\": rpc error: code = NotFound desc = could not find container \"72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c\": container with ID starting with 72378212d036c07fc0e1baf03ba3bd030691b3d906e9f81d92f1c42fabfd079c not found: ID does not exist"
Oct 31 02:10:25.139497 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:25.139422    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt" podStartSLOduration=2.139405812 podStartE2EDuration="2.139405812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:25.139379414 +0000 UTC m=+3946.440335992" watchObservedRunningTime="2026-10-31 02:10:25.139405812 +0000 UTC m=+3946.440362390"
Oct 31 02:10:25.251020 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:25.250988    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"]
Oct 31 02:10:25.257478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:25.257454    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5632ea-openshift-default-5fb9d9fb85-w28vc"]
Oct 31 02:10:25.930110 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:25.930078    5419 generic.go:356] "Generic (PLEG): container finished" podID="7eecb47e-7a7c-4fc7-8b21-f016b524912e" containerID="2ab2c44ee9aa53419064d6a4dfdbc699da901199d4b3429aa454252fce56e2f0" exitCode=0
Oct 31 02:10:25.930420 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:25.930170    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt" event={"ID":"7eecb47e-7a7c-4fc7-8b21-f016b524912e","Type":"ContainerDied","Data":"2ab2c44ee9aa53419064d6a4dfdbc699da901199d4b3429aa454252fce56e2f0"}
Oct 31 02:10:26.007892 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:26.007858    5419 patch_prober.go:29] interesting pod/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:10:26.008095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:26.007911    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:10:26.837012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:26.836976    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" path="/var/lib/kubelet/pods/606c61cc-b66d-44a3-adca-c194a4879ea9/volumes"
Oct 31 02:10:27.069581 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.069558    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:27.173571 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.173466    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca\") pod \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") "
Oct 31 02:10:27.173772 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.173616    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca\") pod \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") "
Oct 31 02:10:27.173772 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.173653    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj\" (UniqueName: \"kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj\") pod \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\" (UID: \"7eecb47e-7a7c-4fc7-8b21-f016b524912e\") "
Oct 31 02:10:27.173931 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.173905    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca" pod "7eecb47e-7a7c-4fc7-8b21-f016b524912e" (UID: "7eecb47e-7a7c-4fc7-8b21-f016b524912e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:27.175980 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.175948    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj" pod "7eecb47e-7a7c-4fc7-8b21-f016b524912e" (UID: "7eecb47e-7a7c-4fc7-8b21-f016b524912e"). InnerVolumeSpecName "kube-api-access-bqpdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:27.176103 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.175976    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca" pod "7eecb47e-7a7c-4fc7-8b21-f016b524912e" (UID: "7eecb47e-7a7c-4fc7-8b21-f016b524912e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:27.274661 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.274628    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:27.274661 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.274654    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqpdj\" (UniqueName: \"kubernetes.io/projected/7eecb47e-7a7c-4fc7-8b21-f016b524912e-kube-api-access-bqpdj\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:27.274661 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.274664    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eecb47e-7a7c-4fc7-8b21-f016b524912e-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:27.944035 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.944008    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"
Oct 31 02:10:27.944513 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.944054    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt" event={"ID":"7eecb47e-7a7c-4fc7-8b21-f016b524912e","Type":"ContainerDied","Data":"0d9db81384e7a61ea137e74ab41ee415818713669d69b198799608e5d78014e7"}
Oct 31 02:10:27.944513 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.944107    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d9db81384e7a61ea137e74ab41ee415818713669d69b198799608e5d78014e7"
Oct 31 02:10:27.991892 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:27.991867    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:10:28.234457 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.234431    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 02:10:28.386179 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386146    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386193    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386234    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386257    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386305    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386346    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386382 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386368    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386416    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386441    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:28.386726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386468    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data\") pod \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\" (UID: \"4a75745b-0605-44fa-bc58-7fcc8fd51dfe\") "
Oct 31 02:10:28.386726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386543    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:28.386951 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386780    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:28.386951 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386806    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:28.386951 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386857    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.386951 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386875    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.387376 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.386971    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:28.388672 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.388644    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:28.388793 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.388669    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "kube-api-access-2q4d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:28.388793 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.388762    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:10:28.388793 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.388781    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy" pod "4a75745b-0605-44fa-bc58-7fcc8fd51dfe" (UID: "4a75745b-0605-44fa-bc58-7fcc8fd51dfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:28.487711 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487681    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.487711 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487712    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.487711 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487722    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.487957 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487735    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4d4\" (UniqueName: \"kubernetes.io/projected/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-kube-api-access-2q4d4\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.487957 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487749    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.487957 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487759    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.487957 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.487767    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a75745b-0605-44fa-bc58-7fcc8fd51dfe-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:28.951328 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.951294    5419 generic.go:356] "Generic (PLEG): container finished" podID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerID="6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22" exitCode=0
Oct 31 02:10:28.951771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.951362    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"
Oct 31 02:10:28.951771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.951379    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" event={"ID":"4a75745b-0605-44fa-bc58-7fcc8fd51dfe","Type":"ContainerDied","Data":"6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22"}
Oct 31 02:10:28.951771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.951435    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds" event={"ID":"4a75745b-0605-44fa-bc58-7fcc8fd51dfe","Type":"ContainerDied","Data":"45762e627fa74796cc416fb95703d0fc7af1f44efcbb1df69dac6461c9228a2a"}
Oct 31 02:10:28.951771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.951464    5419 scope.go:122] "RemoveContainer" containerID="6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22"
Oct 31 02:10:28.960870 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.960852    5419 scope.go:122] "RemoveContainer" containerID="742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"
Oct 31 02:10:28.969470 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.969454    5419 scope.go:122] "RemoveContainer" containerID="6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22"
Oct 31 02:10:28.969717 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:28.969700    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22\": container with ID starting with 6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22 not found: ID does not exist" containerID="6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22"
Oct 31 02:10:28.969762 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.969730    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22"} err="failed to get container status \"6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22\": rpc error: code = NotFound desc = could not find container \"6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22\": container with ID starting with 6bf9541d9ca26e4be47babadb7e325e58d858f9e89cdb3616714925dfdecde22 not found: ID does not exist"
Oct 31 02:10:28.969762 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.969746    5419 scope.go:122] "RemoveContainer" containerID="742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"
Oct 31 02:10:28.969992 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:28.969972    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae\": container with ID starting with 742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae not found: ID does not exist" containerID="742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"
Oct 31 02:10:28.970050 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:28.970002    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae"} err="failed to get container status \"742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae\": rpc error: code = NotFound desc = could not find container \"742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae\": container with ID starting with 742a6227fb269b254417d735a1757de60147c6e91fd6261f65f001d19fe9a2ae not found: ID does not exist"
Oct 31 02:10:29.164091 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:29.163992    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"]
Oct 31 02:10:29.213568 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:29.213534    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-5e916d-openshift-default-7f494f97b5-8mrds"]
Oct 31 02:10:30.836669 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:30.836640    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" path="/var/lib/kubelet/pods/4a75745b-0605-44fa-bc58-7fcc8fd51dfe/volumes"
Oct 31 02:10:31.834249 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:31.834215    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"]
Oct 31 02:10:31.943932 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:31.943900    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"]
Oct 31 02:10:32.044658 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:32.044626    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"]
Oct 31 02:10:32.046388 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:32.046366    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d66ffa8d-m5b85"]
Oct 31 02:10:32.836170 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:32.836121    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4314f51-8cae-40ef-a986-942812e3210e" path="/var/lib/kubelet/pods/f4314f51-8cae-40ef-a986-942812e3210e/volumes"
Oct 31 02:10:34.272361 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:34.272325    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"]
Oct 31 02:10:34.378812 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:34.378772    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"]
Oct 31 02:10:36.835423 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:36.835387    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2" podUID="fd6684c2-8011-4447-855c-7671fbc0a984" containerName="payload-processing" containerID="cri-o://b18aa6cdb4bdb0c502ac90a8e63d00266b95274023a456659ed0bdf4b919216a" gracePeriod=25
Oct 31 02:10:36.944309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:36.944264    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerName="payload-pre-processing" containerID="cri-o://622437b3af648ee8321100ade2e04c9d45064f385454ccf9b31299160a777096" gracePeriod=25
Oct 31 02:10:36.999334 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:36.999298    5419 generic.go:356] "Generic (PLEG): container finished" podID="fd6684c2-8011-4447-855c-7671fbc0a984" containerID="b18aa6cdb4bdb0c502ac90a8e63d00266b95274023a456659ed0bdf4b919216a" exitCode=0
Oct 31 02:10:36.999518 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:36.999362    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2" event={"ID":"fd6684c2-8011-4447-855c-7671fbc0a984","Type":"ContainerDied","Data":"b18aa6cdb4bdb0c502ac90a8e63d00266b95274023a456659ed0bdf4b919216a"}
Oct 31 02:10:37.071982 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.071956    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 02:10:37.168726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.168645    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume\") pod \"fd6684c2-8011-4447-855c-7671fbc0a984\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") "
Oct 31 02:10:37.168726 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.168695    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9\" (UniqueName: \"kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9\") pod \"fd6684c2-8011-4447-855c-7671fbc0a984\" (UID: \"fd6684c2-8011-4447-855c-7671fbc0a984\") "
Oct 31 02:10:37.169005 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.168985    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume" pod "fd6684c2-8011-4447-855c-7671fbc0a984" (UID: "fd6684c2-8011-4447-855c-7671fbc0a984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:37.171024 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.170999    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9" pod "fd6684c2-8011-4447-855c-7671fbc0a984" (UID: "fd6684c2-8011-4447-855c-7671fbc0a984"). InnerVolumeSpecName "kube-api-access-qhjz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:37.270378 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.270338    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd6684c2-8011-4447-855c-7671fbc0a984-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:37.270378 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:37.270371    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjz9\" (UniqueName: \"kubernetes.io/projected/fd6684c2-8011-4447-855c-7671fbc0a984-kube-api-access-qhjz9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:38.006313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:38.006281    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"
Oct 31 02:10:38.006754 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:38.006280    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2" event={"ID":"fd6684c2-8011-4447-855c-7671fbc0a984","Type":"ContainerDied","Data":"c6a472b13be5f534ccbc288aa50b2f9464d4c203650059069f9413e431a9ccaa"}
Oct 31 02:10:38.006754 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:38.006415    5419 scope.go:122] "RemoveContainer" containerID="b18aa6cdb4bdb0c502ac90a8e63d00266b95274023a456659ed0bdf4b919216a"
Oct 31 02:10:38.170349 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:38.170316    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"]
Oct 31 02:10:38.219802 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:38.219760    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-00e840-5f5d89756f-ngbn2"]
Oct 31 02:10:38.836154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:38.836112    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6684c2-8011-4447-855c-7671fbc0a984" path="/var/lib/kubelet/pods/fd6684c2-8011-4447-855c-7671fbc0a984/volumes"
Oct 31 02:10:39.272986 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.272924    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh" podUID="e9f66be6-fbda-413f-b547-299a6b16726e" containerName="payload-processing" containerID="cri-o://08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f" gracePeriod=25
Oct 31 02:10:39.380014 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.379974    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf" podUID="914cee20-683f-4545-bc7f-fb81b3813a06" containerName="payload-pre-processing" containerID="cri-o://76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f" gracePeriod=25
Oct 31 02:10:39.521862 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.521839    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 02:10:39.626584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.626562    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 02:10:39.695220 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.695183    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq\" (UniqueName: \"kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq\") pod \"e9f66be6-fbda-413f-b547-299a6b16726e\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") "
Oct 31 02:10:39.695412 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.695364    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume\") pod \"e9f66be6-fbda-413f-b547-299a6b16726e\" (UID: \"e9f66be6-fbda-413f-b547-299a6b16726e\") "
Oct 31 02:10:39.695698 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.695675    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume" pod "e9f66be6-fbda-413f-b547-299a6b16726e" (UID: "e9f66be6-fbda-413f-b547-299a6b16726e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:39.697374 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.697356    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq" pod "e9f66be6-fbda-413f-b547-299a6b16726e" (UID: "e9f66be6-fbda-413f-b547-299a6b16726e"). InnerVolumeSpecName "kube-api-access-585zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:39.796076 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.796034    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h\" (UniqueName: \"kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h\") pod \"914cee20-683f-4545-bc7f-fb81b3813a06\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") "
Oct 31 02:10:39.796076 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.796082    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume\") pod \"914cee20-683f-4545-bc7f-fb81b3813a06\" (UID: \"914cee20-683f-4545-bc7f-fb81b3813a06\") "
Oct 31 02:10:39.796351 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.796332    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-585zq\" (UniqueName: \"kubernetes.io/projected/e9f66be6-fbda-413f-b547-299a6b16726e-kube-api-access-585zq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.796394 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.796358    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f66be6-fbda-413f-b547-299a6b16726e-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.796480 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.796463    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume" pod "914cee20-683f-4545-bc7f-fb81b3813a06" (UID: "914cee20-683f-4545-bc7f-fb81b3813a06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:39.798371 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.798350    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h" pod "914cee20-683f-4545-bc7f-fb81b3813a06" (UID: "914cee20-683f-4545-bc7f-fb81b3813a06"). InnerVolumeSpecName "kube-api-access-wr49h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:39.818669 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.818594    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main" containerID="cri-o://132da53ac792fff1ba7922ec108fc88811610e6c06d0d189bae8310e193729a8" gracePeriod=45
Oct 31 02:10:39.838758 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.838728    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main" containerID="cri-o://37d99237226346e21a412dcd1979efd7cc39eae3f70d134a550d289cd0c5af5c" gracePeriod=45
Oct 31 02:10:39.897655 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.897619    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr49h\" (UniqueName: \"kubernetes.io/projected/914cee20-683f-4545-bc7f-fb81b3813a06-kube-api-access-wr49h\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:39.897655 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:39.897649    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914cee20-683f-4545-bc7f-fb81b3813a06-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.023535 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.023505    5419 generic.go:356] "Generic (PLEG): container finished" podID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerID="132da53ac792fff1ba7922ec108fc88811610e6c06d0d189bae8310e193729a8" exitCode=0
Oct 31 02:10:40.023712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.023575    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" event={"ID":"7cd2af97-bc5d-4af8-abf9-d6363d04233d","Type":"ContainerDied","Data":"132da53ac792fff1ba7922ec108fc88811610e6c06d0d189bae8310e193729a8"}
Oct 31 02:10:40.025370 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.025343    5419 generic.go:356] "Generic (PLEG): container finished" podID="914cee20-683f-4545-bc7f-fb81b3813a06" containerID="76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f" exitCode=0
Oct 31 02:10:40.025482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.025409    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"
Oct 31 02:10:40.025482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.025417    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf" event={"ID":"914cee20-683f-4545-bc7f-fb81b3813a06","Type":"ContainerDied","Data":"76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f"}
Oct 31 02:10:40.025482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.025469    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf" event={"ID":"914cee20-683f-4545-bc7f-fb81b3813a06","Type":"ContainerDied","Data":"4cda3e49fc7dc98515146d46876e574c09498c9fb5b8146cdcf708af426cbb6e"}
Oct 31 02:10:40.025623 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.025502    5419 scope.go:122] "RemoveContainer" containerID="76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f"
Oct 31 02:10:40.027033 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.027011    5419 generic.go:356] "Generic (PLEG): container finished" podID="e9f66be6-fbda-413f-b547-299a6b16726e" containerID="08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f" exitCode=0
Oct 31 02:10:40.027456 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.027065    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"
Oct 31 02:10:40.027456 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.027127    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh" event={"ID":"e9f66be6-fbda-413f-b547-299a6b16726e","Type":"ContainerDied","Data":"08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f"}
Oct 31 02:10:40.027456 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.027175    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh" event={"ID":"e9f66be6-fbda-413f-b547-299a6b16726e","Type":"ContainerDied","Data":"78423caad306252b3ad12843abd42df55ef1143061290bf492221cd74fd8dbfe"}
Oct 31 02:10:40.040611 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.040384    5419 scope.go:122] "RemoveContainer" containerID="76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f"
Oct 31 02:10:40.040611 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.040472    5419 generic.go:356] "Generic (PLEG): container finished" podID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerID="37d99237226346e21a412dcd1979efd7cc39eae3f70d134a550d289cd0c5af5c" exitCode=0
Oct 31 02:10:40.040611 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.040541    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" event={"ID":"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f","Type":"ContainerDied","Data":"37d99237226346e21a412dcd1979efd7cc39eae3f70d134a550d289cd0c5af5c"}
Oct 31 02:10:40.042115 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:40.042088    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f\": container with ID starting with 76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f not found: ID does not exist" containerID="76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f"
Oct 31 02:10:40.042267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.042128    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f"} err="failed to get container status \"76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f\": rpc error: code = NotFound desc = could not find container \"76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f\": container with ID starting with 76cb2de5ef4988d13b5cbb49d989ffe003c42b696f8b53e01d89a9b8461bb20f not found: ID does not exist"
Oct 31 02:10:40.042267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.042169    5419 scope.go:122] "RemoveContainer" containerID="08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f"
Oct 31 02:10:40.053671 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.053649    5419 scope.go:122] "RemoveContainer" containerID="08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f"
Oct 31 02:10:40.054162 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:40.054120    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f\": container with ID starting with 08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f not found: ID does not exist" containerID="08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f"
Oct 31 02:10:40.054252 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.054175    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f"} err="failed to get container status \"08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f\": rpc error: code = NotFound desc = could not find container \"08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f\": container with ID starting with 08d27b931a5a04b4974da089be3263f695c34acd027fa0d6b66fa0056465916f not found: ID does not exist"
Oct 31 02:10:40.076381 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.076320    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 02:10:40.108664 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.108642    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 02:10:40.188183 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.188150    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"]
Oct 31 02:10:40.200090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200058    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home\") pod \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") "
Oct 31 02:10:40.200291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200117    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc\" (UniqueName: \"kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc\") pod \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") "
Oct 31 02:10:40.200291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200187    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir\") pod \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") "
Oct 31 02:10:40.200291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200236    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm\") pod \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") "
Oct 31 02:10:40.200291 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200282    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs\") pod \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") "
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200307    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache\") pod \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") "
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200363    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home" pod "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" (UID: "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200371    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm\") pod \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") "
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200428    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir\") pod \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") "
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200480    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache\") pod \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") "
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200520    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home\") pod \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") "
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200529    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir" pod "7cd2af97-bc5d-4af8-abf9-d6363d04233d" (UID: "7cd2af97-bc5d-4af8-abf9-d6363d04233d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.200577 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200557    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs\") pod \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\" (UID: \"7cd2af97-bc5d-4af8-abf9-d6363d04233d\") "
Oct 31 02:10:40.200992 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200596    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9\" (UniqueName: \"kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9\") pod \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\" (UID: \"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f\") "
Oct 31 02:10:40.200992 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200700    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache" pod "7cd2af97-bc5d-4af8-abf9-d6363d04233d" (UID: "7cd2af97-bc5d-4af8-abf9-d6363d04233d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.200992 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200697    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir" pod "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" (UID: "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.200992 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200753    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache" pod "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" (UID: "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.200992 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.200909    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home" pod "7cd2af97-bc5d-4af8-abf9-d6363d04233d" (UID: "7cd2af97-bc5d-4af8-abf9-d6363d04233d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.201301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.201038    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.201301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.201057    5419 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-home\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.201301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.201069    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.201301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.201083    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.201301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.201097    5419 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tmp-dir\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.201301 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.201110    5419 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-model-cache\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.203111 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.203054    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9" pod "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" (UID: "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f"). InnerVolumeSpecName "kube-api-access-64cx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:40.203235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.203129    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs" pod "7cd2af97-bc5d-4af8-abf9-d6363d04233d" (UID: "7cd2af97-bc5d-4af8-abf9-d6363d04233d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:40.203235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.203169    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm" pod "7cd2af97-bc5d-4af8-abf9-d6363d04233d" (UID: "7cd2af97-bc5d-4af8-abf9-d6363d04233d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.203235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.203187    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc" pod "7cd2af97-bc5d-4af8-abf9-d6363d04233d" (UID: "7cd2af97-bc5d-4af8-abf9-d6363d04233d"). InnerVolumeSpecName "kube-api-access-2cxtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:40.203235 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.203207    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm" pod "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" (UID: "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:10:40.203424 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.203406    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs" pod "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" (UID: "cd0c6e95-4f05-4c05-9bcb-f5c82d05358f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:40.237691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.237659    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-71bc98-79f49cdff7-fb4jh"]
Oct 31 02:10:40.301895 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.301866    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxtc\" (UniqueName: \"kubernetes.io/projected/7cd2af97-bc5d-4af8-abf9-d6363d04233d-kube-api-access-2cxtc\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.301895 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.301894    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd2af97-bc5d-4af8-abf9-d6363d04233d-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.302449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.301905    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.302449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.301913    5419 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-dshm\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.302449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.301921    5419 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd2af97-bc5d-4af8-abf9-d6363d04233d-tls-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.302449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.301930    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64cx9\" (UniqueName: \"kubernetes.io/projected/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f-kube-api-access-64cx9\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:40.460955 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.460833    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"]
Oct 31 02:10:40.466527 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.466498    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-71bc98-775bccfc95-tgzcf"]
Oct 31 02:10:40.836645 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.836611    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914cee20-683f-4545-bc7f-fb81b3813a06" path="/var/lib/kubelet/pods/914cee20-683f-4545-bc7f-fb81b3813a06/volumes"
Oct 31 02:10:40.836979 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:40.836966    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f66be6-fbda-413f-b547-299a6b16726e" path="/var/lib/kubelet/pods/e9f66be6-fbda-413f-b547-299a6b16726e/volumes"
Oct 31 02:10:41.047830 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.047800    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"
Oct 31 02:10:41.048039 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.047843    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w" event={"ID":"cd0c6e95-4f05-4c05-9bcb-f5c82d05358f","Type":"ContainerDied","Data":"1cb5d2c4a3d681b7da972354c04b622c53ef35ea2a47bd9708f0a5af9a9be133"}
Oct 31 02:10:41.048039 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.047896    5419 scope.go:122] "RemoveContainer" containerID="37d99237226346e21a412dcd1979efd7cc39eae3f70d134a550d289cd0c5af5c"
Oct 31 02:10:41.049385 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.049352    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"
Oct 31 02:10:41.049475 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.049389    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4" event={"ID":"7cd2af97-bc5d-4af8-abf9-d6363d04233d","Type":"ContainerDied","Data":"0204193a4a3572e4d18875c09f14418eb87e74d7601ccb4e8037ca13d3cefaeb"}
Oct 31 02:10:41.057912 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.057890    5419 scope.go:122] "RemoveContainer" containerID="132da53ac792fff1ba7922ec108fc88811610e6c06d0d189bae8310e193729a8"
Oct 31 02:10:41.211495 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.211406    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"]
Oct 31 02:10:41.260809 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.260779    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/distinct-w2-00e840-kserve-7865645bcd-7vlp4"]
Oct 31 02:10:41.321584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.321544    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"]
Oct 31 02:10:41.323610 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:41.323589    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-00e840/simulator-w2-00e840-kserve-65466f45d-grg2w"]
Oct 31 02:10:42.009233 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:42.009197    5419 scope.go:122] "RemoveContainer" containerID="5243f75c3fbe40415fa2fca1435bcfa140500d10b094d307232bad13781895f8"
Oct 31 02:10:42.835437 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:42.835407    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" path="/var/lib/kubelet/pods/7cd2af97-bc5d-4af8-abf9-d6363d04233d/volumes"
Oct 31 02:10:42.835809 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:42.835795    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" path="/var/lib/kubelet/pods/cd0c6e95-4f05-4c05-9bcb-f5c82d05358f/volumes"
Oct 31 02:10:43.353681 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:43.353644    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:10:44.581479 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:10:44.581443    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:10:46.799270 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799234    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"]
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799606    5419 cpu_manager.go:414] "containerMap: removing container" podUID="914cee20-683f-4545-bc7f-fb81b3813a06" containerName="payload-pre-processing"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799626    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914cee20-683f-4545-bc7f-fb81b3813a06" containerName="payload-pre-processing"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799638    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7eecb47e-7a7c-4fc7-8b21-f016b524912e" containerName="revoke-keys"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799643    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eecb47e-7a7c-4fc7-8b21-f016b524912e" containerName="revoke-keys"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799651    5419 cpu_manager.go:414] "containerMap: removing container" podUID="fd6684c2-8011-4447-855c-7671fbc0a984" containerName="payload-processing"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799656    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6684c2-8011-4447-855c-7671fbc0a984" containerName="payload-processing"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799667    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799672    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799678    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799683    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799691    5419 cpu_manager.go:414] "containerMap: removing container" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799696    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799705    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e9f66be6-fbda-413f-b547-299a6b16726e" containerName="payload-processing"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799710    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f66be6-fbda-413f-b547-299a6b16726e" containerName="payload-processing"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799717    5419 cpu_manager.go:414] "containerMap: removing container" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy"
Oct 31 02:10:46.799712 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799722    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799732    5419 cpu_manager.go:414] "containerMap: removing container" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799738    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799750    5419 cpu_manager.go:414] "containerMap: removing container" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799754    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799806    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799812    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f66be6-fbda-413f-b547-299a6b16726e" containerName="payload-processing"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799817    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799826    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="606c61cc-b66d-44a3-adca-c194a4879ea9" containerName="istio-proxy"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799833    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd2af97-bc5d-4af8-abf9-d6363d04233d" containerName="main"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799841    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="914cee20-683f-4545-bc7f-fb81b3813a06" containerName="payload-pre-processing"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799847    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eecb47e-7a7c-4fc7-8b21-f016b524912e" containerName="revoke-keys"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799852    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6684c2-8011-4447-855c-7671fbc0a984" containerName="payload-processing"
Oct 31 02:10:46.800309 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.799859    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0c6e95-4f05-4c05-9bcb-f5c82d05358f" containerName="main"
Oct 31 02:10:46.802789 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.802773    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:46.857199 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.857169    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"]
Oct 31 02:10:46.857598 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.857585    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a75745b-0605-44fa-bc58-7fcc8fd51dfe" containerName="istio-proxy"
Oct 31 02:10:46.860813 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.860787    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:46.904560 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.904525    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:10:46.904560 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.904549    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:10:46.904731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.904534    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:10:46.904731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.904656    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:10:46.906490 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.906473    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"]
Oct 31 02:10:46.956394 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.956353    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"]
Oct 31 02:10:46.961872 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.961845    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsmg\" (UniqueName: \"kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:46.962069 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.961897    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlcg\" (UniqueName: \"kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:46.962069 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.961915    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:46.962069 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.961982    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:46.962323 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.962102    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:46.962323 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:46.962225    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.063125 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063040    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.063125 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063106    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsmg\" (UniqueName: \"kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.063413 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063155    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlcg\" (UniqueName: \"kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.063413 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063174    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.063413 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063194    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.063413 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063228    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.063868 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063838    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.063996 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.063916    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.065703 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.065681    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.065831 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.065819    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.132786 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.132754    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlcg\" (UniqueName: \"kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg\") pod \"maas-api-revoke-sub-d4cdf4bf-7hrzs\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.135845 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.135810    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsmg\" (UniqueName: \"kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg\") pod \"maas-api-revoke-sub-5cee7d2f-rwjlr\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.174758 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.174721    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:47.415694 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.415268    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:47.547279 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.547256    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"]
Oct 31 02:10:47.549383 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:10:47.549354    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18c9edd_8f96_47d1_870f_d4430713e707.slice/crio-0a953068e9060c884dfab39ef5dec0c9ab90b35846fa06414da9aefccadac1d1 WatchSource:0}: Error finding container 0a953068e9060c884dfab39ef5dec0c9ab90b35846fa06414da9aefccadac1d1: Status 404 returned error can't find the container with id 0a953068e9060c884dfab39ef5dec0c9ab90b35846fa06414da9aefccadac1d1
Oct 31 02:10:47.791056 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:47.791034    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"]
Oct 31 02:10:47.793274 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:10:47.793246    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f595983_4582_463d_8cea_55ecf1cc55f5.slice/crio-76d9a3cbc90a3a113fa08cdb50232f18a29944913fed7cb92daffc43c04a02ae WatchSource:0}: Error finding container 76d9a3cbc90a3a113fa08cdb50232f18a29944913fed7cb92daffc43c04a02ae: Status 404 returned error can't find the container with id 76d9a3cbc90a3a113fa08cdb50232f18a29944913fed7cb92daffc43c04a02ae
Oct 31 02:10:48.092668 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:48.092639    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs" event={"ID":"0f595983-4582-463d-8cea-55ecf1cc55f5","Type":"ContainerStarted","Data":"76d9a3cbc90a3a113fa08cdb50232f18a29944913fed7cb92daffc43c04a02ae"}
Oct 31 02:10:48.094129 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:48.094106    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr" event={"ID":"b18c9edd-8f96-47d1-870f-d4430713e707","Type":"ContainerStarted","Data":"cbcd7ef39891591fbef435fabde0d1606e19157fd5cabbde86f231e2c3a9e61c"}
Oct 31 02:10:48.094283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:48.094160    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr" event={"ID":"b18c9edd-8f96-47d1-870f-d4430713e707","Type":"ContainerStarted","Data":"0a953068e9060c884dfab39ef5dec0c9ab90b35846fa06414da9aefccadac1d1"}
Oct 31 02:10:48.207831 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:48.207779    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr" podStartSLOduration=2.207763516 podStartE2EDuration="2.207763516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:48.206341575 +0000 UTC m=+3969.507298153" watchObservedRunningTime="2026-10-31 02:10:48.207763516 +0000 UTC m=+3969.508720093"
Oct 31 02:10:49.049840 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:49.049747    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"]
Oct 31 02:10:49.099171 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:49.099122    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-71bc98-dzmbt"]
Oct 31 02:10:49.102608 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:49.102580    5419 generic.go:356] "Generic (PLEG): container finished" podID="0f595983-4582-463d-8cea-55ecf1cc55f5" containerID="dadf737eb4af5920a44cd1f7916303a2e5fcffd6713036a8cffadd0b481eb9a2" exitCode=0
Oct 31 02:10:49.102757 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:49.102700    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs" event={"ID":"0f595983-4582-463d-8cea-55ecf1cc55f5","Type":"ContainerDied","Data":"dadf737eb4af5920a44cd1f7916303a2e5fcffd6713036a8cffadd0b481eb9a2"}
Oct 31 02:10:49.104316 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:49.104292    5419 generic.go:356] "Generic (PLEG): container finished" podID="b18c9edd-8f96-47d1-870f-d4430713e707" containerID="cbcd7ef39891591fbef435fabde0d1606e19157fd5cabbde86f231e2c3a9e61c" exitCode=0
Oct 31 02:10:49.104406 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:49.104379    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr" event={"ID":"b18c9edd-8f96-47d1-870f-d4430713e707","Type":"ContainerDied","Data":"cbcd7ef39891591fbef435fabde0d1606e19157fd5cabbde86f231e2c3a9e61c"}
Oct 31 02:10:50.250830 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.250805    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:50.278477 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.278453    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:50.299761 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.299733    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg\" (UniqueName: \"kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg\") pod \"0f595983-4582-463d-8cea-55ecf1cc55f5\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") "
Oct 31 02:10:50.299903 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.299777    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca\") pod \"0f595983-4582-463d-8cea-55ecf1cc55f5\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") "
Oct 31 02:10:50.299903 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.299805    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg\" (UniqueName: \"kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg\") pod \"b18c9edd-8f96-47d1-870f-d4430713e707\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") "
Oct 31 02:10:50.299903 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.299876    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca\") pod \"0f595983-4582-463d-8cea-55ecf1cc55f5\" (UID: \"0f595983-4582-463d-8cea-55ecf1cc55f5\") "
Oct 31 02:10:50.300068 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.299910    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca\") pod \"b18c9edd-8f96-47d1-870f-d4430713e707\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") "
Oct 31 02:10:50.300068 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.299983    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca\") pod \"b18c9edd-8f96-47d1-870f-d4430713e707\" (UID: \"b18c9edd-8f96-47d1-870f-d4430713e707\") "
Oct 31 02:10:50.300256 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.300130    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca" pod "0f595983-4582-463d-8cea-55ecf1cc55f5" (UID: "0f595983-4582-463d-8cea-55ecf1cc55f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:50.300385 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.300358    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca" pod "b18c9edd-8f96-47d1-870f-d4430713e707" (UID: "b18c9edd-8f96-47d1-870f-d4430713e707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:50.302211 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.302187    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg" pod "0f595983-4582-463d-8cea-55ecf1cc55f5" (UID: "0f595983-4582-463d-8cea-55ecf1cc55f5"). InnerVolumeSpecName "kube-api-access-9hlcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:50.302211 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.302191    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca" pod "0f595983-4582-463d-8cea-55ecf1cc55f5" (UID: "0f595983-4582-463d-8cea-55ecf1cc55f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:50.302375 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.302222    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg" pod "b18c9edd-8f96-47d1-870f-d4430713e707" (UID: "b18c9edd-8f96-47d1-870f-d4430713e707"). InnerVolumeSpecName "kube-api-access-lmsmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:50.302375 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.302295    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca" pod "b18c9edd-8f96-47d1-870f-d4430713e707" (UID: "b18c9edd-8f96-47d1-870f-d4430713e707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:10:50.401662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.401573    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlcg\" (UniqueName: \"kubernetes.io/projected/0f595983-4582-463d-8cea-55ecf1cc55f5-kube-api-access-9hlcg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.401662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.401613    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.401662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.401624    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsmg\" (UniqueName: \"kubernetes.io/projected/b18c9edd-8f96-47d1-870f-d4430713e707-kube-api-access-lmsmg\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.401662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.401633    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f595983-4582-463d-8cea-55ecf1cc55f5-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.401662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.401642    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.401662 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.401650    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b18c9edd-8f96-47d1-870f-d4430713e707-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:50.838754 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:50.838712    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eecb47e-7a7c-4fc7-8b21-f016b524912e" path="/var/lib/kubelet/pods/7eecb47e-7a7c-4fc7-8b21-f016b524912e/volumes"
Oct 31 02:10:51.117591 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:51.117504    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs"
Oct 31 02:10:51.117591 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:51.117527    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cdf4bf-7hrzs" event={"ID":"0f595983-4582-463d-8cea-55ecf1cc55f5","Type":"ContainerDied","Data":"76d9a3cbc90a3a113fa08cdb50232f18a29944913fed7cb92daffc43c04a02ae"}
Oct 31 02:10:51.117591 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:51.117578    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d9a3cbc90a3a113fa08cdb50232f18a29944913fed7cb92daffc43c04a02ae"
Oct 31 02:10:51.118916 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:51.118899    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr"
Oct 31 02:10:51.119041 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:51.118952    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cee7d2f-rwjlr" event={"ID":"b18c9edd-8f96-47d1-870f-d4430713e707","Type":"ContainerDied","Data":"0a953068e9060c884dfab39ef5dec0c9ab90b35846fa06414da9aefccadac1d1"}
Oct 31 02:10:51.119041 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:51.118998    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a953068e9060c884dfab39ef5dec0c9ab90b35846fa06414da9aefccadac1d1"
Oct 31 02:10:53.353732 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:53.353682    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:10:53.768987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:53.768945    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:10:56.369494 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:56.369454    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"]
Oct 31 02:10:56.369998 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:56.369659    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" containerID="cri-o://92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982" gracePeriod=30
Oct 31 02:10:57.158925 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.158892    5419 generic.go:356] "Generic (PLEG): container finished" podID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerID="622437b3af648ee8321100ade2e04c9d45064f385454ccf9b31299160a777096" exitCode=0
Oct 31 02:10:57.159154 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.158969    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" event={"ID":"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3","Type":"ContainerDied","Data":"622437b3af648ee8321100ade2e04c9d45064f385454ccf9b31299160a777096"}
Oct 31 02:10:57.190164 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.190124    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 02:10:57.263936 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.263899    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2\" (UniqueName: \"kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2\") pod \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") "
Oct 31 02:10:57.264102 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.263995    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume\") pod \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\" (UID: \"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3\") "
Oct 31 02:10:57.264488 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.264461    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume" pod "15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" (UID: "15c9a7d8-a8d6-42d9-abbe-96d39977d4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:10:57.266408 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.266379    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2" pod "15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" (UID: "15c9a7d8-a8d6-42d9-abbe-96d39977d4e3"). InnerVolumeSpecName "kube-api-access-tfpv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:10:57.365245 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.365200    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfpv2\" (UniqueName: \"kubernetes.io/projected/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-kube-api-access-tfpv2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:57.365245 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.365244    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:10:57.377924 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.377892    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"]
Oct 31 02:10:57.378464 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378444    5419 cpu_manager.go:414] "containerMap: removing container" podUID="0f595983-4582-463d-8cea-55ecf1cc55f5" containerName="revoke-keys"
Oct 31 02:10:57.378544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378467    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f595983-4582-463d-8cea-55ecf1cc55f5" containerName="revoke-keys"
Oct 31 02:10:57.378544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378497    5419 cpu_manager.go:414] "containerMap: removing container" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerName="payload-pre-processing"
Oct 31 02:10:57.378544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378507    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerName="payload-pre-processing"
Oct 31 02:10:57.378544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378531    5419 cpu_manager.go:414] "containerMap: removing container" podUID="b18c9edd-8f96-47d1-870f-d4430713e707" containerName="revoke-keys"
Oct 31 02:10:57.378544 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378538    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18c9edd-8f96-47d1-870f-d4430713e707" containerName="revoke-keys"
Oct 31 02:10:57.378844 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378645    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f595983-4582-463d-8cea-55ecf1cc55f5" containerName="revoke-keys"
Oct 31 02:10:57.378844 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378659    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" containerName="payload-pre-processing"
Oct 31 02:10:57.378844 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.378671    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18c9edd-8f96-47d1-870f-d4430713e707" containerName="revoke-keys"
Oct 31 02:10:57.381884 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.381864    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.435963 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.435929    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:10:57.436245 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.436218    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:10:57.436520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.436489    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:10:57.436635 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.436618    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:10:57.438934 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.438911    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"]
Oct 31 02:10:57.466370 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.466340    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.466498 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.466380    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.466565 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.466503    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkddj\" (UniqueName: \"kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.567362 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.567283    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.567362 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.567327    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.567564 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.567385    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkddj\" (UniqueName: \"kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.568020 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.567992    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.569830 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.569806    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.623335 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.623309    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkddj\" (UniqueName: \"kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj\") pod \"maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:57.694558 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:57.694524    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:10:58.065766 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.065725    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"]
Oct 31 02:10:58.068268 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:10:58.068234    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d935f6_d934_495f_afcf_8cbb157569a8.slice/crio-c132404b341d8fd663b1af7c97b5e5b1a4f543654f37402d6cc757263d037c99 WatchSource:0}: Error finding container c132404b341d8fd663b1af7c97b5e5b1a4f543654f37402d6cc757263d037c99: Status 404 returned error can't find the container with id c132404b341d8fd663b1af7c97b5e5b1a4f543654f37402d6cc757263d037c99
Oct 31 02:10:58.165955 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.165924    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls" event={"ID":"79d935f6-d934-495f-afcf-8cbb157569a8","Type":"ContainerStarted","Data":"c132404b341d8fd663b1af7c97b5e5b1a4f543654f37402d6cc757263d037c99"}
Oct 31 02:10:58.167478 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.167445    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6" event={"ID":"15c9a7d8-a8d6-42d9-abbe-96d39977d4e3","Type":"ContainerDied","Data":"22f224491139bbe568afc08c29c13dbedcd0bf28e9ea0c8801448fa435f0cb91"}
Oct 31 02:10:58.167642 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.167497    5419 scope.go:122] "RemoveContainer" containerID="622437b3af648ee8321100ade2e04c9d45064f385454ccf9b31299160a777096"
Oct 31 02:10:58.167642 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.167459    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"
Oct 31 02:10:58.279288 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.279260    5419 patch_prober.go:29] interesting pod/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 02:10:58.279449 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.279304    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 02:10:58.332026 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.331939    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"]
Oct 31 02:10:58.333500 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.333479    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-00e840-59cf68bb67-6m8q6"]
Oct 31 02:10:58.844670 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:58.844639    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c9a7d8-a8d6-42d9-abbe-96d39977d4e3" path="/var/lib/kubelet/pods/15c9a7d8-a8d6-42d9-abbe-96d39977d4e3/volumes"
Oct 31 02:10:59.175553 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:59.175462    5419 generic.go:356] "Generic (PLEG): container finished" podID="79d935f6-d934-495f-afcf-8cbb157569a8" containerID="f3ff6524bb022b9e7dd8f46b3d60ec5b23f4a24233310d18683cc0d189aa7735" exitCode=0
Oct 31 02:10:59.175553 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:10:59.175539    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls" event={"ID":"79d935f6-d934-495f-afcf-8cbb157569a8","Type":"ContainerDied","Data":"f3ff6524bb022b9e7dd8f46b3d60ec5b23f4a24233310d18683cc0d189aa7735"}
Oct 31 02:11:00.315549 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.315526    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:11:00.392926 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.392895    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca\") pod \"79d935f6-d934-495f-afcf-8cbb157569a8\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") "
Oct 31 02:11:00.393118 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.392938    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj\" (UniqueName: \"kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj\") pod \"79d935f6-d934-495f-afcf-8cbb157569a8\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") "
Oct 31 02:11:00.393118 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.392979    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca\") pod \"79d935f6-d934-495f-afcf-8cbb157569a8\" (UID: \"79d935f6-d934-495f-afcf-8cbb157569a8\") "
Oct 31 02:11:00.393356 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.393333    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca" pod "79d935f6-d934-495f-afcf-8cbb157569a8" (UID: "79d935f6-d934-495f-afcf-8cbb157569a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:00.395209 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.395182    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca" pod "79d935f6-d934-495f-afcf-8cbb157569a8" (UID: "79d935f6-d934-495f-afcf-8cbb157569a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:00.395313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.395210    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj" pod "79d935f6-d934-495f-afcf-8cbb157569a8" (UID: "79d935f6-d934-495f-afcf-8cbb157569a8"). InnerVolumeSpecName "kube-api-access-xkddj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:00.481328 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.481296    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"]
Oct 31 02:11:00.482846 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.482821    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-00e840-sbg9h"]
Oct 31 02:11:00.494161 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.494120    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:00.494283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.494164    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkddj\" (UniqueName: \"kubernetes.io/projected/79d935f6-d934-495f-afcf-8cbb157569a8-kube-api-access-xkddj\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:00.494283 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.494181    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d935f6-d934-495f-afcf-8cbb157569a8-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:00.835795 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:00.835763    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa354fec-85b8-45d1-b6f6-40499a06317a" path="/var/lib/kubelet/pods/aa354fec-85b8-45d1-b6f6-40499a06317a/volumes"
Oct 31 02:11:01.188702 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.188617    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"
Oct 31 02:11:01.188702 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.188650    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls" event={"ID":"79d935f6-d934-495f-afcf-8cbb157569a8","Type":"ContainerDied","Data":"c132404b341d8fd663b1af7c97b5e5b1a4f543654f37402d6cc757263d037c99"}
Oct 31 02:11:01.188902 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.188717    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c132404b341d8fd663b1af7c97b5e5b1a4f543654f37402d6cc757263d037c99"
Oct 31 02:11:01.621247 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.621219    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 02:11:01.705885 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.705850    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.705905    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.705958    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706034    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706095 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706092    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706375 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706169    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706375 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706181    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:01.706375 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706201    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706375 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706243    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706600 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706383    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:01.706600 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706524    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket\") pod \"01660afd-9883-4a1b-9056-2885f954648b\" (UID: \"01660afd-9883-4a1b-9056-2885f954648b\") "
Oct 31 02:11:01.706747 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706725    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:01.706907 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706890    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.706981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706899    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:01.706981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706915    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.706981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706930    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.706981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.706964    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:01.712298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.712265    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:01.712298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.712263    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:11:01.712298 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.712289    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:01.721368 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.721340    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2" pod "01660afd-9883-4a1b-9056-2885f954648b" (UID: "01660afd-9883-4a1b-9056-2885f954648b"). InnerVolumeSpecName "kube-api-access-sp8n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:01.808267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.808191    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01660afd-9883-4a1b-9056-2885f954648b-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.808267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.808219    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01660afd-9883-4a1b-9056-2885f954648b-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.808267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.808230    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.808267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.808239    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp8n2\" (UniqueName: \"kubernetes.io/projected/01660afd-9883-4a1b-9056-2885f954648b-kube-api-access-sp8n2\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.808267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.808252    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:01.808267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:01.808260    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01660afd-9883-4a1b-9056-2885f954648b-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:02.196158 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.196056    5419 generic.go:356] "Generic (PLEG): container finished" podID="01660afd-9883-4a1b-9056-2885f954648b" containerID="92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982" exitCode=0
Oct 31 02:11:02.196158 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.196111    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"
Oct 31 02:11:02.196381 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.196153    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" event={"ID":"01660afd-9883-4a1b-9056-2885f954648b","Type":"ContainerDied","Data":"92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982"}
Oct 31 02:11:02.196381 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.196209    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl" event={"ID":"01660afd-9883-4a1b-9056-2885f954648b","Type":"ContainerDied","Data":"f3ca3bee1227e5a4b667c8fd3345c89709d42ea08e40b789ea780b6215d05de4"}
Oct 31 02:11:02.196381 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.196221    5419 scope.go:122] "RemoveContainer" containerID="92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982"
Oct 31 02:11:02.206578 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.206559    5419 scope.go:122] "RemoveContainer" containerID="dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"
Oct 31 02:11:02.214797 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.214781    5419 scope.go:122] "RemoveContainer" containerID="92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982"
Oct 31 02:11:02.215074 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:11:02.215054    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982\": container with ID starting with 92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982 not found: ID does not exist" containerID="92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982"
Oct 31 02:11:02.215123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.215084    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982"} err="failed to get container status \"92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982\": rpc error: code = NotFound desc = could not find container \"92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982\": container with ID starting with 92a8c5cf3be7df4e030e2006170419aedfc5d1387c4de3e782f83461fbb0a982 not found: ID does not exist"
Oct 31 02:11:02.215123 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.215101    5419 scope.go:122] "RemoveContainer" containerID="dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"
Oct 31 02:11:02.215392 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:11:02.215375    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451\": container with ID starting with dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451 not found: ID does not exist" containerID="dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"
Oct 31 02:11:02.215434 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.215400    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451"} err="failed to get container status \"dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451\": rpc error: code = NotFound desc = could not find container \"dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451\": container with ID starting with dc00ac9eb9a5bcc7e52727fdf64237979ddb7ad2b926a926594adf322d725451 not found: ID does not exist"
Oct 31 02:11:02.357523 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.357479    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"]
Oct 31 02:11:02.407419 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.407386    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-71bc98-openshift-default-84786b6cdc-5h7hl"]
Oct 31 02:11:02.835890 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:02.835856    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01660afd-9883-4a1b-9056-2885f954648b" path="/var/lib/kubelet/pods/01660afd-9883-4a1b-9056-2885f954648b/volumes"
Oct 31 02:11:08.846319 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:08.846286    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"]
Oct 31 02:11:09.706771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.706736    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"]
Oct 31 02:11:09.707086 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707074    5419 cpu_manager.go:414] "containerMap: removing container" podUID="79d935f6-d934-495f-afcf-8cbb157569a8" containerName="revoke-keys"
Oct 31 02:11:09.707131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707088    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d935f6-d934-495f-afcf-8cbb157569a8" containerName="revoke-keys"
Oct 31 02:11:09.707131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707114    5419 cpu_manager.go:414] "containerMap: removing container" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy"
Oct 31 02:11:09.707131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707119    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy"
Oct 31 02:11:09.707131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707131    5419 cpu_manager.go:414] "containerMap: removing container" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy"
Oct 31 02:11:09.707131 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707150    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy"
Oct 31 02:11:09.707327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707205    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy"
Oct 31 02:11:09.707327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707215    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d935f6-d934-495f-afcf-8cbb157569a8" containerName="revoke-keys"
Oct 31 02:11:09.707327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.707225    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="01660afd-9883-4a1b-9056-2885f954648b" containerName="istio-proxy"
Oct 31 02:11:09.711771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.711755    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.765372 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.765345    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"]
Oct 31 02:11:09.766981 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.766947    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 02:11:09.767091 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.767009    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zzj8k\""
Oct 31 02:11:09.768926 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.768902    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.769340 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.769293    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"]
Oct 31 02:11:09.769719 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.769703    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 02:11:09.769830 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.769755    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 02:11:09.780065 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.780043    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.780198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.780080    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjgbd\" (UniqueName: \"kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.780198 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.780105    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.825536 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.825505    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"]
Oct 31 02:11:09.881748 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.881713    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.882267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.881766    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5n28\" (UniqueName: \"kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.882267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.881798    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjgbd\" (UniqueName: \"kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.882267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.881833    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.882267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.881940    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.882267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.882070    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.882521 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.882380    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.884474 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.884453    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.938029 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.938005    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjgbd\" (UniqueName: \"kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd\") pod \"maas-api-revoke-sub-eb230c29-6n8dm\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:09.983521 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.983493    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5n28\" (UniqueName: \"kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.983684 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.983538    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.983684 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.983559    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.984159 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.984124    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:09.985917 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:09.985896    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:10.024126 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:10.024100    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:10.040178 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:10.040128    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5n28\" (UniqueName: \"kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28\") pod \"maas-api-revoke-sub-bc9bac2c-nnb8w\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:10.080711 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:10.080678    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:10.397426 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:10.397401    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"]
Oct 31 02:11:10.399792 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:11:10.399765    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31270269_1aee_44c1_b61e_a27fa3f05245.slice/crio-c1b5a7dc25bdb2b6fd4830c4b51da3d022eb49c0e41e56982e746d584ddb487a WatchSource:0}: Error finding container c1b5a7dc25bdb2b6fd4830c4b51da3d022eb49c0e41e56982e746d584ddb487a: Status 404 returned error can't find the container with id c1b5a7dc25bdb2b6fd4830c4b51da3d022eb49c0e41e56982e746d584ddb487a
Oct 31 02:11:10.458474 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:10.458401    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"]
Oct 31 02:11:10.460557 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:11:10.460529    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd981f87a_416f_4647_9d86_5237b5c31448.slice/crio-aeb709a3d68b9d8037bf66770c2422410a2d0ea3464add0c30c8fcf2a13b941b WatchSource:0}: Error finding container aeb709a3d68b9d8037bf66770c2422410a2d0ea3464add0c30c8fcf2a13b941b: Status 404 returned error can't find the container with id aeb709a3d68b9d8037bf66770c2422410a2d0ea3464add0c30c8fcf2a13b941b
Oct 31 02:11:11.250197 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:11.250133    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w" event={"ID":"d981f87a-416f-4647-9d86-5237b5c31448","Type":"ContainerStarted","Data":"c64660f3c63e86d63ed9fd11050289f9acb922a9c7ece5486ff356f870bd2b69"}
Oct 31 02:11:11.250651 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:11.250212    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w" event={"ID":"d981f87a-416f-4647-9d86-5237b5c31448","Type":"ContainerStarted","Data":"aeb709a3d68b9d8037bf66770c2422410a2d0ea3464add0c30c8fcf2a13b941b"}
Oct 31 02:11:11.251892 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:11.251866    5419 generic.go:356] "Generic (PLEG): container finished" podID="31270269-1aee-44c1-b61e-a27fa3f05245" containerID="6b6b0f943fc52f6a124788e619027634bf3a111122507a645e7afa0c8638ea38" exitCode=0
Oct 31 02:11:11.252025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:11.251910    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm" event={"ID":"31270269-1aee-44c1-b61e-a27fa3f05245","Type":"ContainerDied","Data":"6b6b0f943fc52f6a124788e619027634bf3a111122507a645e7afa0c8638ea38"}
Oct 31 02:11:11.252025 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:11.251957    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm" event={"ID":"31270269-1aee-44c1-b61e-a27fa3f05245","Type":"ContainerStarted","Data":"c1b5a7dc25bdb2b6fd4830c4b51da3d022eb49c0e41e56982e746d584ddb487a"}
Oct 31 02:11:11.363911 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:11.363858    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w" podStartSLOduration=2.363843869 podStartE2EDuration="2.363843869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:11:11.362927831 +0000 UTC m=+3992.663884409" watchObservedRunningTime="2026-10-31 02:11:11.363843869 +0000 UTC m=+3992.664800458"
Oct 31 02:11:12.260326 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.260288    5419 generic.go:356] "Generic (PLEG): container finished" podID="d981f87a-416f-4647-9d86-5237b5c31448" containerID="c64660f3c63e86d63ed9fd11050289f9acb922a9c7ece5486ff356f870bd2b69" exitCode=0
Oct 31 02:11:12.260725 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.260369    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w" event={"ID":"d981f87a-416f-4647-9d86-5237b5c31448","Type":"ContainerDied","Data":"c64660f3c63e86d63ed9fd11050289f9acb922a9c7ece5486ff356f870bd2b69"}
Oct 31 02:11:12.395802 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.395777    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:12.506203 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.506170    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca\") pod \"31270269-1aee-44c1-b61e-a27fa3f05245\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") "
Oct 31 02:11:12.506378 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.506254    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca\") pod \"31270269-1aee-44c1-b61e-a27fa3f05245\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") "
Oct 31 02:11:12.506378 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.506325    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd\" (UniqueName: \"kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd\") pod \"31270269-1aee-44c1-b61e-a27fa3f05245\" (UID: \"31270269-1aee-44c1-b61e-a27fa3f05245\") "
Oct 31 02:11:12.506576 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.506557    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca" pod "31270269-1aee-44c1-b61e-a27fa3f05245" (UID: "31270269-1aee-44c1-b61e-a27fa3f05245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:12.509295 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.509022    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd" pod "31270269-1aee-44c1-b61e-a27fa3f05245" (UID: "31270269-1aee-44c1-b61e-a27fa3f05245"). InnerVolumeSpecName "kube-api-access-cjgbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:12.511562 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.511485    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca" pod "31270269-1aee-44c1-b61e-a27fa3f05245" (UID: "31270269-1aee-44c1-b61e-a27fa3f05245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:12.607344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.607316    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjgbd\" (UniqueName: \"kubernetes.io/projected/31270269-1aee-44c1-b61e-a27fa3f05245-kube-api-access-cjgbd\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.607344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.607341    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:12.607344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:12.607351    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31270269-1aee-44c1-b61e-a27fa3f05245-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:13.266897 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.266863    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm" event={"ID":"31270269-1aee-44c1-b61e-a27fa3f05245","Type":"ContainerDied","Data":"c1b5a7dc25bdb2b6fd4830c4b51da3d022eb49c0e41e56982e746d584ddb487a"}
Oct 31 02:11:13.267437 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.266929    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1b5a7dc25bdb2b6fd4830c4b51da3d022eb49c0e41e56982e746d584ddb487a"
Oct 31 02:11:13.267437 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.266877    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb230c29-6n8dm"
Oct 31 02:11:13.395804 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.395782    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:13.515159 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.515116    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca\") pod \"d981f87a-416f-4647-9d86-5237b5c31448\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") "
Oct 31 02:11:13.515325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.515206    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca\") pod \"d981f87a-416f-4647-9d86-5237b5c31448\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") "
Oct 31 02:11:13.515325 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.515283    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28\" (UniqueName: \"kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28\") pod \"d981f87a-416f-4647-9d86-5237b5c31448\" (UID: \"d981f87a-416f-4647-9d86-5237b5c31448\") "
Oct 31 02:11:13.515561 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.515541    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca" pod "d981f87a-416f-4647-9d86-5237b5c31448" (UID: "d981f87a-416f-4647-9d86-5237b5c31448"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:13.517584 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.517521    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca" pod "d981f87a-416f-4647-9d86-5237b5c31448" (UID: "d981f87a-416f-4647-9d86-5237b5c31448"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 02:11:13.517691 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.517586    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28" pod "d981f87a-416f-4647-9d86-5237b5c31448" (UID: "d981f87a-416f-4647-9d86-5237b5c31448"). InnerVolumeSpecName "kube-api-access-q5n28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:13.616314 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.616277    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-xks-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:13.616314 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.616308    5419 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d981f87a-416f-4647-9d86-5237b5c31448-maas-api-openshift-ca\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:13.616529 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.616323    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5n28\" (UniqueName: \"kubernetes.io/projected/d981f87a-416f-4647-9d86-5237b5c31448-kube-api-access-q5n28\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:13.846899 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:13.846813    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerName="payload-processing" containerID="cri-o://7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362" gracePeriod=25
Oct 31 02:11:14.273971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:14.273939    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w"
Oct 31 02:11:14.273971 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:14.273947    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9bac2c-nnb8w" event={"ID":"d981f87a-416f-4647-9d86-5237b5c31448","Type":"ContainerDied","Data":"aeb709a3d68b9d8037bf66770c2422410a2d0ea3464add0c30c8fcf2a13b941b"}
Oct 31 02:11:14.274470 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:14.273990    5419 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb709a3d68b9d8037bf66770c2422410a2d0ea3464add0c30c8fcf2a13b941b"
Oct 31 02:11:14.582027 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:11:14.581945    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:11:16.220300 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:16.220264    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:11:24.834158 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:24.833996    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:11:26.220415 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:26.220376    5419 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 02:11:26.515991 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:26.515903    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"]
Oct 31 02:11:26.565327 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:26.565289    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-baacee-zpjls"]
Oct 31 02:11:26.836640 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:26.836566    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d935f6-d934-495f-afcf-8cbb157569a8" path="/var/lib/kubelet/pods/79d935f6-d934-495f-afcf-8cbb157569a8/volumes"
Oct 31 02:11:33.045693 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:33.045656    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"]
Oct 31 02:11:33.094908 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:33.094878    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7b04c-gkqgh"]
Oct 31 02:11:33.217883 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:33.217849    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"]
Oct 31 02:11:33.218184 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:33.218131    5419 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy" containerID="cri-o://811463d03de9539ef2b80a46f03f469be89517a71d956c0232beecc4e3d4f691" gracePeriod=30
Oct 31 02:11:34.086963 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.086941    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 02:11:34.212090 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.212018    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq\" (UniqueName: \"kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq\") pod \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") "
Oct 31 02:11:34.212278 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.212104    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume\") pod \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\" (UID: \"e7ade45e-da1d-4a36-ba56-228c267a8fa4\") "
Oct 31 02:11:34.212620 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.212583    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume" pod "e7ade45e-da1d-4a36-ba56-228c267a8fa4" (UID: "e7ade45e-da1d-4a36-ba56-228c267a8fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:34.214675 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.214653    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq" pod "e7ade45e-da1d-4a36-ba56-228c267a8fa4" (UID: "e7ade45e-da1d-4a36-ba56-228c267a8fa4"). InnerVolumeSpecName "kube-api-access-kxcgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:34.313027 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.312991    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxcgq\" (UniqueName: \"kubernetes.io/projected/e7ade45e-da1d-4a36-ba56-228c267a8fa4-kube-api-access-kxcgq\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:34.313027 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.313021    5419 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7ade45e-da1d-4a36-ba56-228c267a8fa4-plugins-config-volume\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:34.388988 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.388951    5419 generic.go:356] "Generic (PLEG): container finished" podID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerID="7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362" exitCode=0
Oct 31 02:11:34.389189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.389006    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"
Oct 31 02:11:34.389189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.389040    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" event={"ID":"e7ade45e-da1d-4a36-ba56-228c267a8fa4","Type":"ContainerDied","Data":"7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362"}
Oct 31 02:11:34.389189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.389097    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4" event={"ID":"e7ade45e-da1d-4a36-ba56-228c267a8fa4","Type":"ContainerDied","Data":"84783eb91c51b82c9d156ddfe93660509a8d34337e86cedfd1798f20e39da0fa"}
Oct 31 02:11:34.389189 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.389120    5419 scope.go:122] "RemoveContainer" containerID="7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362"
Oct 31 02:11:34.398581 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.398566    5419 scope.go:122] "RemoveContainer" containerID="7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362"
Oct 31 02:11:34.398853 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:11:34.398836    5419 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362\": container with ID starting with 7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362 not found: ID does not exist" containerID="7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362"
Oct 31 02:11:34.398933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.398862    5419 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362"} err="failed to get container status \"7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362\": rpc error: code = NotFound desc = could not find container \"7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362\": container with ID starting with 7d0fb58ae40a683d61906383b6060fb42e13c329890854f4ab81ebd0e14d8362 not found: ID does not exist"
Oct 31 02:11:34.550374 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.550339    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"]
Oct 31 02:11:34.554437 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.554410    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-baacee-7c84b68df5-hp4s4"]
Oct 31 02:11:34.587784 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.587758    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65fdc4ffcd-ncpr6_779ab764-2e40-47e1-b415-0f7307c9d002/authorino/0.log"
Oct 31 02:11:34.836009 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.835938    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7fd4ca-ab96-4ddd-803b-782f50ef6166" path="/var/lib/kubelet/pods/9c7fd4ca-ab96-4ddd-803b-782f50ef6166/volumes"
Oct 31 02:11:34.836321 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.836308    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" path="/var/lib/kubelet/pods/e7ade45e-da1d-4a36-ba56-228c267a8fa4/volumes"
Oct 31 02:11:34.989829 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:34.989797    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v8knf_3f1e8c92-0009-4922-824d-162960b28060/manager/0.log"
Oct 31 02:11:37.128903 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:37.128880    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vlvpt_93d3c753-4f46-49ca-9f45-bfdfd1679571/limitador/0.log"
Oct 31 02:11:38.416731 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.416697    5419 generic.go:356] "Generic (PLEG): container finished" podID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerID="811463d03de9539ef2b80a46f03f469be89517a71d956c0232beecc4e3d4f691" exitCode=0
Oct 31 02:11:38.417184 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.416768    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" event={"ID":"cb3bb1ca-6e4a-465f-bb37-18141137d234","Type":"ContainerDied","Data":"811463d03de9539ef2b80a46f03f469be89517a71d956c0232beecc4e3d4f691"}
Oct 31 02:11:38.417184 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.416848    5419 scope.go:122] "RemoveContainer" containerID="c2a9da38006f89c7e4d3e7bc8f0803ab6ab38dae1d395d3776d5f8255ee3a02f"
Oct 31 02:11:38.469753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.469732    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 02:11:38.552430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.552355    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.552430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.552394    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.552430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.552423    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.552739 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.552453    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.552739 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.552636    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:38.554738 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.554708    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:38.554857 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.554747    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 02:11:38.554857 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.554766    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "kube-api-access-lqfnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 02:11:38.652949 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.652915    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.652949 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.652955    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.653240 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.652980    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.653240 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653002    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.653240 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653031    5419 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy\") pod \"cb3bb1ca-6e4a-465f-bb37-18141137d234\" (UID: \"cb3bb1ca-6e4a-465f-bb37-18141137d234\") "
Oct 31 02:11:38.653411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653319    5419 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-podinfo\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.653411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653342    5419 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfnh\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-kube-api-access-lqfnh\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.653411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653358    5419 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.653411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653358    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:38.653411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653372    5419 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-token\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.653411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653382    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:38.653633 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653417    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 02:11:38.653633 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.653444    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:38.655393 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.655365    5419 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy" pod "cb3bb1ca-6e4a-465f-bb37-18141137d234" (UID: "cb3bb1ca-6e4a-465f-bb37-18141137d234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 02:11:38.754278 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.754246    5419 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3bb1ca-6e4a-465f-bb37-18141137d234-istiod-ca-cert\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.754278 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.754273    5419 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-data\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.754278 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.754282    5419 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-workload-certs\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.754520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.754291    5419 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-credential-socket\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:38.754520 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:38.754300    5419 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3bb1ca-6e4a-465f-bb37-18141137d234-istio-envoy\") on node \"ip-10-0-140-164.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 02:11:39.423684 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:39.423655    5419 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"
Oct 31 02:11:39.423684 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:39.423665    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh" event={"ID":"cb3bb1ca-6e4a-465f-bb37-18141137d234","Type":"ContainerDied","Data":"d119d0f1b38f5cd1a9f171eb88982b1fb5cbc78fed14e8bc6f12eab502db8026"}
Oct 31 02:11:39.424267 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:39.423735    5419 scope.go:122] "RemoveContainer" containerID="811463d03de9539ef2b80a46f03f469be89517a71d956c0232beecc4e3d4f691"
Oct 31 02:11:39.639541 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:39.639463    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"]
Oct 31 02:11:39.642654 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:39.642621    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-baacee-openshift-default-57f845d5bf-8mqsh"]
Oct 31 02:11:40.834821 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:40.834788    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" path="/var/lib/kubelet/pods/cb3bb1ca-6e4a-465f-bb37-18141137d234/volumes"
Oct 31 02:11:41.344601 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:41.344570    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65fdc4ffcd-ncpr6_779ab764-2e40-47e1-b415-0f7307c9d002/authorino/0.log"
Oct 31 02:11:42.084104 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:42.084068    5419 scope.go:122] "RemoveContainer" containerID="9c8bdff65742ac5bda369407da257b1856480286b211d5af55b0fd88e92c0044"
Oct 31 02:11:42.092851 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:42.092828    5419 scope.go:122] "RemoveContainer" containerID="3e8c93024717b44665591399906f1a8a0dd720b77d4f7336eb97cf3af654d5b5"
Oct 31 02:11:44.055673 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:44.055636    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"]
Oct 31 02:11:44.105023 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:44.104995    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a83a034f-p6gkr"]
Oct 31 02:11:44.581985 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:11:44.581952    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:11:44.835707 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:44.835633    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99941559-bf52-4b0d-a930-e7f9282007ef" path="/var/lib/kubelet/pods/99941559-bf52-4b0d-a930-e7f9282007ef/volumes"
Oct 31 02:11:44.928792 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:44.928762    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-vlvpt"]
Oct 31 02:11:55.759582 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:55.759549    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0392c9d6-h28c2_b0f7be2b-866d-4e43-8ee7-b9d711600a64/revoke-keys/0.log"
Oct 31 02:11:55.985943 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:55.985912    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-063e7c83-bt5dx_e8ee14a8-4fc0-4696-a2e3-554a9b9f8086/revoke-keys/0.log"
Oct 31 02:11:56.214918 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:56.214842    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2fcf5072-wbtfm_d0d4fa9b-870c-4aeb-b7d5-4a1a1f275955/revoke-keys/0.log"
Oct 31 02:11:56.460324 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:56.460293    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3ae8608c-s7fzv_54915945-ccd8-4e05-b8f9-256d85f0f112/revoke-keys/0.log"
Oct 31 02:11:56.708130 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:56.708103    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5cee7d2f-rwjlr_b18c9edd-8f96-47d1-870f-d4430713e707/revoke-keys/0.log"
Oct 31 02:11:56.940576 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:56.940549    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-69e80696-5x79q_2df90395-e58f-4c87-b2c1-b7f9201720e1/revoke-keys/0.log"
Oct 31 02:11:57.154616 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:57.154585    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c94b8bd-lljwp_5d2a64d6-4066-4f81-82f9-50046d991664/revoke-keys/0.log"
Oct 31 02:11:57.382223 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:57.382195    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7d7915c6-hx5rl_2855d516-7749-424b-8bb1-de991fc737cf/revoke-keys/0.log"
Oct 31 02:11:57.615313 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:57.615286    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b484d7ec-r8zkt_aba786b8-404f-4add-8a3c-b7ca53bca85d/revoke-keys/0.log"
Oct 31 02:11:57.850355 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:57.850323    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bc9bac2c-nnb8w_d981f87a-416f-4647-9d86-5237b5c31448/revoke-keys/0.log"
Oct 31 02:11:58.070933 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:58.070902    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d4cdf4bf-7hrzs_0f595983-4582-463d-8cea-55ecf1cc55f5/revoke-keys/0.log"
Oct 31 02:11:58.295810 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:11:58.295780    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eb230c29-6n8dm_31270269-1aee-44c1-b61e-a27fa3f05245/revoke-keys/0.log"
Oct 31 02:12:00.127867 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:00.127836    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65fdc4ffcd-ncpr6_779ab764-2e40-47e1-b415-0f7307c9d002/authorino/0.log"
Oct 31 02:12:00.521461 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:00.521429    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v8knf_3f1e8c92-0009-4922-824d-162960b28060/manager/0.log"
Oct 31 02:12:14.580708 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:12:14.580668    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:12:22.365850 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:22.365772    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vlvpt_93d3c753-4f46-49ca-9f45-bfdfd1679571/limitador/0.log"
Oct 31 02:12:23.897597 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:23.897568    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"
Oct 31 02:12:24.448700 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:24.448670    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-sgs8x_3410a6d1-0d32-4001-8588-fca0dea3fbaf/kube-auth-proxy/0.log"
Oct 31 02:12:25.286899 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:25.286874    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rsgqk_1c3606b3-e3d1-4bd8-bf2e-424c31fe3756/payload-pre-processing/0.log"
Oct 31 02:12:26.706374 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:26.706344    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-wl4pk_05fa0135-c2dc-48b5-80bb-0e48e9eca65c/main/0.log"
Oct 31 02:12:27.266511 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:27.266477    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7rlk4_e87dac0d-52ca-4169-986f-95ea2aa75916/main/0.log"
Oct 31 02:12:27.749174 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:27.748498    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hwv2f_78e23333-17c5-4811-ade2-829d0e2d5548/main/0.log"
Oct 31 02:12:28.448792 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:28.448762    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-pt7dc_222d5179-79a4-4cfd-b925-6a899bb608ba/main/0.log"
Oct 31 02:12:34.393998 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:34.393957    5419 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal" podUID="7360d5bb713e28adcb32faa1b6003ade" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 02:12:42.134112 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:42.134079    5419 scope.go:122] "RemoveContainer" containerID="d0093d1d68e69c728d5114529138b7fdce50817bf867c50002f1c911d804ce1c"
Oct 31 02:12:42.143083 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:42.143059    5419 scope.go:122] "RemoveContainer" containerID="9e71316acfb6c0ca39bf1bde63b82c5464b4b947ba21b7c27ad5f1c6a949494a"
Oct 31 02:12:42.151454 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:42.151431    5419 scope.go:122] "RemoveContainer" containerID="8870ab99f5603f50dfdd3b583350a9f7d95ac39bac411fdccfe78857782da0f0"
Oct 31 02:12:44.580638 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:12:44.580598    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:12:45.051614 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:45.051582    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"]
Oct 31 02:12:45.100627 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:45.100597    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b484d7ec-r8zkt"]
Oct 31 02:12:46.835275 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:46.835244    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba786b8-404f-4add-8a3c-b7ca53bca85d" path="/var/lib/kubelet/pods/aba786b8-404f-4add-8a3c-b7ca53bca85d/volumes"
Oct 31 02:12:48.933318 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933279    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sz4w8/must-gather-z8jkz"]
Oct 31 02:12:48.933754 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933737    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy"
Oct 31 02:12:48.933811 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933758    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy"
Oct 31 02:12:48.933811 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933778    5419 cpu_manager.go:414] "containerMap: removing container" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerName="payload-processing"
Oct 31 02:12:48.933811 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933784    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerName="payload-processing"
Oct 31 02:12:48.933811 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933796    5419 cpu_manager.go:414] "containerMap: removing container" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy"
Oct 31 02:12:48.933811 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933804    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933829    5419 cpu_manager.go:414] "containerMap: removing container" podUID="31270269-1aee-44c1-b61e-a27fa3f05245" containerName="revoke-keys"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933834    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31270269-1aee-44c1-b61e-a27fa3f05245" containerName="revoke-keys"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933841    5419 cpu_manager.go:414] "containerMap: removing container" podUID="d981f87a-416f-4647-9d86-5237b5c31448" containerName="revoke-keys"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933847    5419 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d981f87a-416f-4647-9d86-5237b5c31448" containerName="revoke-keys"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933930    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="d981f87a-416f-4647-9d86-5237b5c31448" containerName="revoke-keys"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933942    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933950    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ade45e-da1d-4a36-ba56-228c267a8fa4" containerName="payload-processing"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933961    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3bb1ca-6e4a-465f-bb37-18141137d234" containerName="istio-proxy"
Oct 31 02:12:48.934012 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.933971    5419 memory_manager.go:365] "RemoveStaleState removing state" podUID="31270269-1aee-44c1-b61e-a27fa3f05245" containerName="revoke-keys"
Oct 31 02:12:48.937354 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.937340    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:48.993182 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:48.993161    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sz4w8/must-gather-z8jkz"]
Oct 31 02:12:49.034675 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.034640    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqh9\" (UniqueName: \"kubernetes.io/projected/c33a4069-60f2-439c-ac44-853ec4ed3824-kube-api-access-pfqh9\") pod \"must-gather-z8jkz\" (UID: \"c33a4069-60f2-439c-ac44-853ec4ed3824\") " pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.034675 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.034675    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c33a4069-60f2-439c-ac44-853ec4ed3824-must-gather-output\") pod \"must-gather-z8jkz\" (UID: \"c33a4069-60f2-439c-ac44-853ec4ed3824\") " pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.040655 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.040634    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sz4w8\"/\"openshift-service-ca.crt\""
Oct 31 02:12:49.040750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.040692    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sz4w8\"/\"kube-root-ca.crt\""
Oct 31 02:12:49.040750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.040726    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sz4w8\"/\"default-dockercfg-c2bwf\""
Oct 31 02:12:49.053057 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.053027    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"]
Oct 31 02:12:49.054344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.054326    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"]
Oct 31 02:12:49.092859 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.092832    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c94b8bd-lljwp"]
Oct 31 02:12:49.101777 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.101753    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-063e7c83-bt5dx"]
Oct 31 02:12:49.135524 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.135501    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqh9\" (UniqueName: \"kubernetes.io/projected/c33a4069-60f2-439c-ac44-853ec4ed3824-kube-api-access-pfqh9\") pod \"must-gather-z8jkz\" (UID: \"c33a4069-60f2-439c-ac44-853ec4ed3824\") " pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.135656 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.135530    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c33a4069-60f2-439c-ac44-853ec4ed3824-must-gather-output\") pod \"must-gather-z8jkz\" (UID: \"c33a4069-60f2-439c-ac44-853ec4ed3824\") " pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.135838 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.135819    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c33a4069-60f2-439c-ac44-853ec4ed3824-must-gather-output\") pod \"must-gather-z8jkz\" (UID: \"c33a4069-60f2-439c-ac44-853ec4ed3824\") " pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.191612 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.191547    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqh9\" (UniqueName: \"kubernetes.io/projected/c33a4069-60f2-439c-ac44-853ec4ed3824-kube-api-access-pfqh9\") pod \"must-gather-z8jkz\" (UID: \"c33a4069-60f2-439c-ac44-853ec4ed3824\") " pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.250595 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.250562    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sz4w8/must-gather-z8jkz"
Oct 31 02:12:49.620335 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.620305    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sz4w8/must-gather-z8jkz"]
Oct 31 02:12:49.623190 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:12:49.623157    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc33a4069_60f2_439c_ac44_853ec4ed3824.slice/crio-1e15647e54c28255e8397775cf8ebb52fc2ecb58ccce2d3b1680780d673b309c WatchSource:0}: Error finding container 1e15647e54c28255e8397775cf8ebb52fc2ecb58ccce2d3b1680780d673b309c: Status 404 returned error can't find the container with id 1e15647e54c28255e8397775cf8ebb52fc2ecb58ccce2d3b1680780d673b309c
Oct 31 02:12:49.624961 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.624945    5419 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 02:12:49.793852 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:49.793822    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz4w8/must-gather-z8jkz" event={"ID":"c33a4069-60f2-439c-ac44-853ec4ed3824","Type":"ContainerStarted","Data":"1e15647e54c28255e8397775cf8ebb52fc2ecb58ccce2d3b1680780d673b309c"}
Oct 31 02:12:50.835940 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:50.835906    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2a64d6-4066-4f81-82f9-50046d991664" path="/var/lib/kubelet/pods/5d2a64d6-4066-4f81-82f9-50046d991664/volumes"
Oct 31 02:12:50.836509 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:50.836488    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ee14a8-4fc0-4696-a2e3-554a9b9f8086" path="/var/lib/kubelet/pods/e8ee14a8-4fc0-4696-a2e3-554a9b9f8086/volumes"
Oct 31 02:12:56.045066 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:56.045024    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"]
Oct 31 02:12:56.094122 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:56.094077    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e80696-5x79q"]
Oct 31 02:12:56.837727 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:56.837693    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df90395-e58f-4c87-b2c1-b7f9201720e1" path="/var/lib/kubelet/pods/2df90395-e58f-4c87-b2c1-b7f9201720e1/volumes"
Oct 31 02:12:57.845969 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:57.845934    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz4w8/must-gather-z8jkz" event={"ID":"c33a4069-60f2-439c-ac44-853ec4ed3824","Type":"ContainerStarted","Data":"5d318d77423ff73f1772bde2cab8bb92b56f1770188a838fbd4cbfd24f4c787d"}
Oct 31 02:12:57.846411 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:57.845980    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sz4w8/must-gather-z8jkz" event={"ID":"c33a4069-60f2-439c-ac44-853ec4ed3824","Type":"ContainerStarted","Data":"cb8f51ac989e48500c620afdc0a09aa004c8c17ae42698bbff5eb5ca6b659f19"}
Oct 31 02:12:57.966611 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:57.966051    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-sz4w8/must-gather-z8jkz" podStartSLOduration=2.556232311 podStartE2EDuration="9.966031683s" totalImagesPullingTime="7.409799372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:57.964428562 +0000 UTC m=+4099.265385164" watchObservedRunningTime="2026-10-31 02:12:57.966031683 +0000 UTC m=+4099.266988261"
Oct 31 02:12:58.661685 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:12:58.661657    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tmx2h_ef840aa5-db4c-4422-a2d4-702a5c8bdcb0/global-pull-secret-syncer/0.log"
Oct 31 02:13:01.603837 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:01.603809    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-d9bhz_1c9dd849-9bfc-44bf-8eba-b8a08677c824/konnectivity-agent/0.log"
Oct 31 02:13:02.774688 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:02.774654    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-164.us-west-1.compute.internal_7360d5bb713e28adcb32faa1b6003ade/haproxy/0.log"
Oct 31 02:13:14.580631 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:13:14.580592    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:13:18.563519 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:18.563487    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65fdc4ffcd-ncpr6_779ab764-2e40-47e1-b415-0f7307c9d002/authorino/0.log"
Oct 31 02:13:18.900016 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:18.899943    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v8knf_3f1e8c92-0009-4922-824d-162960b28060/manager/0.log"
Oct 31 02:13:21.098217 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:21.098190    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vlvpt_93d3c753-4f46-49ca-9f45-bfdfd1679571/limitador/0.log"
Oct 31 02:13:22.640827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:22.640769    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"
Oct 31 02:13:22.928125 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:22.928046    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"
Oct 31 02:13:22.967804 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:22.967780    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-sgs8x_3410a6d1-0d32-4001-8588-fca0dea3fbaf/kube-auth-proxy/0.log"
Oct 31 02:13:23.248858 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:23.248826    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-sgs8x_3410a6d1-0d32-4001-8588-fca0dea3fbaf/kube-auth-proxy/0.log"
Oct 31 02:13:23.620402 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:23.620326    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rsgqk_1c3606b3-e3d1-4bd8-bf2e-424c31fe3756/payload-pre-processing/0.log"
Oct 31 02:13:23.893833 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:23.893766    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rsgqk_1c3606b3-e3d1-4bd8-bf2e-424c31fe3756/payload-pre-processing/0.log"
Oct 31 02:13:24.230672 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:24.230641    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/alertmanager/0.log"
Oct 31 02:13:24.644921 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:24.644847    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/config-reloader/0.log"
Oct 31 02:13:25.057782 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:25.057752    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/kube-rbac-proxy-web/0.log"
Oct 31 02:13:25.471741 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:25.471715    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/kube-rbac-proxy/0.log"
Oct 31 02:13:25.889952 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:25.889874    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/kube-rbac-proxy-metric/0.log"
Oct 31 02:13:26.304756 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:26.304725    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/prom-label-proxy/0.log"
Oct 31 02:13:26.721455 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:26.721429    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cd6977b9-1cad-406c-b35f-4da0ba739d03/init-config-reloader/0.log"
Oct 31 02:13:28.273617 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.273582    5419 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"]
Oct 31 02:13:28.277207 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.277189    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.336178 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.336120    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"]
Oct 31 02:13:28.380836 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.380810    5419 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-frc7g\"/\"default-dockercfg-lp57d\""
Oct 31 02:13:28.380836 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.380810    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-frc7g\"/\"openshift-service-ca.crt\""
Oct 31 02:13:28.380836 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.380810    5419 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-frc7g\"/\"kube-root-ca.crt\""
Oct 31 02:13:28.390276 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.390243    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-proc\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.390430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.390282    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-podres\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.390430 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.390406    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-sys\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.390557 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.390442    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbbm\" (UniqueName: \"kubernetes.io/projected/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-kube-api-access-6dbbm\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.390557 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.390514    5419 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-lib-modules\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.490978 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.490942    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-proc\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.490978 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.490981    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-podres\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491014    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-sys\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491016    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-proc\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491048    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-podres\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491063    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-sys\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491061    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbbm\" (UniqueName: \"kubernetes.io/projected/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-kube-api-access-6dbbm\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491111    5419 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-lib-modules\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.491289 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.491226    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-lib-modules\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.547482 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.547416    5419 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbbm\" (UniqueName: \"kubernetes.io/projected/fa6e4585-ef1a-4a06-b44a-63e2206d4ecc-kube-api-access-6dbbm\") pod \"perf-node-gather-daemonset-6m9x7\" (UID: \"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc\") " pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.588791 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.588764    5419 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:28.962779 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:28.962699    5419 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"]
Oct 31 02:13:28.966801 ip-10-0-140-164 kubenswrapper[5419]: W1031 02:13:28.966764    5419 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfa6e4585_ef1a_4a06_b44a_63e2206d4ecc.slice/crio-8b52950a7119dcd2791913dc568553534fba28ea628fc218aef946ae48c26827 WatchSource:0}: Error finding container 8b52950a7119dcd2791913dc568553534fba28ea628fc218aef946ae48c26827: Status 404 returned error can't find the container with id 8b52950a7119dcd2791913dc568553534fba28ea628fc218aef946ae48c26827
Oct 31 02:13:29.024516 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:29.024483    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7" event={"ID":"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc","Type":"ContainerStarted","Data":"8b52950a7119dcd2791913dc568553534fba28ea628fc218aef946ae48c26827"}
Oct 31 02:13:29.248196 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:29.248152    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65fdc4ffcd-ncpr6_779ab764-2e40-47e1-b415-0f7307c9d002/authorino/0.log"
Oct 31 02:13:29.374546 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:29.374508    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v8knf_3f1e8c92-0009-4922-824d-162960b28060/manager/0.log"
Oct 31 02:13:29.676987 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:29.676907    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65fdc4ffcd-ncpr6_779ab764-2e40-47e1-b415-0f7307c9d002/authorino/0.log"
Oct 31 02:13:29.701771 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:29.701739    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbjh9_43cb7dd9-ef71-43ac-8631-84d0ee80468a/node-exporter/0.log"
Oct 31 02:13:29.789265 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:29.789240    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-v8knf_3f1e8c92-0009-4922-824d-162960b28060/manager/0.log"
Oct 31 02:13:30.031017 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:30.030983    5419 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7" event={"ID":"fa6e4585-ef1a-4a06-b44a-63e2206d4ecc","Type":"ContainerStarted","Data":"1b762343d016b31fecde0cb6aa3730177399885ae3a97c2aad991b56549b12f1"}
Oct 31 02:13:30.031203 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:30.031115    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:30.113877 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:30.113844    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbjh9_43cb7dd9-ef71-43ac-8631-84d0ee80468a/kube-rbac-proxy/0.log"
Oct 31 02:13:30.146687 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:30.146639    5419 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7" podStartSLOduration=2.146627533 podStartE2EDuration="2.146627533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:30.145408189 +0000 UTC m=+4131.446364766" watchObservedRunningTime="2026-10-31 02:13:30.146627533 +0000 UTC m=+4131.447584100"
Oct 31 02:13:30.571553 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:30.571528    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fbjh9_43cb7dd9-ef71-43ac-8631-84d0ee80468a/init-textfile/0.log"
Oct 31 02:13:30.848029 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:30.847946    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vlvpt_93d3c753-4f46-49ca-9f45-bfdfd1679571/limitador/0.log"
Oct 31 02:13:31.306624 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:31.306594    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-vlvpt_93d3c753-4f46-49ca-9f45-bfdfd1679571/limitador/0.log"
Oct 31 02:13:36.047612 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:36.047583    5419 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-frc7g/perf-node-gather-daemonset-6m9x7"
Oct 31 02:13:37.832786 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:37.832750    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-w4xxz_71865382-46e6-4f6a-bb1f-9b5406439186/prometheus-operator/0.log"
Oct 31 02:13:38.287505 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:38.287476    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-w4xxz_71865382-46e6-4f6a-bb1f-9b5406439186/kube-rbac-proxy/0.log"
Oct 31 02:13:39.125948 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:39.125870    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-757fd9cbf7-46qpv_b865ce7c-d25a-4154-a2cd-3136b2937143/telemeter-client/0.log"
Oct 31 02:13:39.539750 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:39.539706    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-757fd9cbf7-46qpv_b865ce7c-d25a-4154-a2cd-3136b2937143/reload/0.log"
Oct 31 02:13:39.953685 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:39.953615    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-757fd9cbf7-46qpv_b865ce7c-d25a-4154-a2cd-3136b2937143/kube-rbac-proxy/0.log"
Oct 31 02:13:42.177827 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:42.177794    5419 scope.go:122] "RemoveContainer" containerID="e0269fbceb91de0a87711b3094b5cfe5405aa96292aa27ecca13c296e8a10dae"
Oct 31 02:13:42.187924 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:42.187893    5419 scope.go:122] "RemoveContainer" containerID="b37d7c768ceb495db2dd4003fc2bfbcb220a4c6c7499c516ddf03d5654c6a55b"
Oct 31 02:13:42.198179 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:42.198157    5419 scope.go:122] "RemoveContainer" containerID="33a809ac93061e4f63156a6133b98169c806786a16bea012c86dd54e5c962f40"
Oct 31 02:13:42.206317 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:42.206284    5419 scope.go:122] "RemoveContainer" containerID="5f4e7c42eb7b75a48aadb33f0542d4da57ad71b7db7f7aebdeff244c17111dd0"
Oct 31 02:13:44.581166 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:13:44.581111    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:13:49.208498 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:49.208417    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-hbjp8_7ba21b02-c733-4fd0-b735-8d6cbc2927f4/manager/0.log"
Oct 31 02:13:49.402492 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:49.402465    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-hbjp8_7ba21b02-c733-4fd0-b735-8d6cbc2927f4/copy-manifests/0.log"
Oct 31 02:13:50.096753 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:50.096720    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-hbjp8_7ba21b02-c733-4fd0-b735-8d6cbc2927f4/manager/0.log"
Oct 31 02:13:50.291407 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:50.291379    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-559fd7ff56-hbjp8_7ba21b02-c733-4fd0-b735-8d6cbc2927f4/copy-manifests/0.log"
Oct 31 02:13:50.806440 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:50.806409    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-9p7q6_111a48d2-3e5c-4c68-8215-98b0cfc5b301/download-server/0.log"
Oct 31 02:13:58.072822 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:58.072790    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"
Oct 31 02:13:58.400164 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:58.400077    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-sgs8x_3410a6d1-0d32-4001-8588-fca0dea3fbaf/kube-auth-proxy/0.log"
Oct 31 02:13:58.970618 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:58.970579    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"
Oct 31 02:13:59.042508 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:59.042481    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rsgqk_1c3606b3-e3d1-4bd8-bf2e-424c31fe3756/payload-pre-processing/0.log"
Oct 31 02:13:59.320671 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:59.320606    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-sgs8x_3410a6d1-0d32-4001-8588-fca0dea3fbaf/kube-auth-proxy/0.log"
Oct 31 02:13:59.957061 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:13:59.957032    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rsgqk_1c3606b3-e3d1-4bd8-bf2e-424c31fe3756/payload-pre-processing/0.log"
Oct 31 02:14:00.801882 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:00.801844    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zx8f4_96abcc08-0354-41d9-baec-af53dd804fdd/dns/0.log"
Oct 31 02:14:01.214556 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:01.214480    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zx8f4_96abcc08-0354-41d9-baec-af53dd804fdd/kube-rbac-proxy/0.log"
Oct 31 02:14:02.475635 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:02.475607    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qctlt_aa5bbf79-871a-4f48-beea-cb3bf34fbe64/dns-node-resolver/0.log"
Oct 31 02:14:05.702344 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:05.702313    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cmxkm_de59f004-1680-492e-88f9-3278ed575e50/node-ca/0.log"
Oct 31 02:14:12.020015 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:12.019980    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"
Oct 31 02:14:13.678939 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:13.678909    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff59956fc-sgs8x_3410a6d1-0d32-4001-8588-fca0dea3fbaf/kube-auth-proxy/0.log"
Oct 31 02:14:14.582428 ip-10-0-140-164 kubenswrapper[5419]: E1031 02:14:14.582387    5419 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-btnvg" containerName="extractor"
Oct 31 02:14:14.938794 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:14.938709    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rsgqk_1c3606b3-e3d1-4bd8-bf2e-424c31fe3756/payload-pre-processing/0.log"
Oct 31 02:14:19.972659 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:19.972628    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dnvvs_198bdfe6-2b3f-4426-a0cc-6568d22b139d/serve-healthcheck-canary/0.log"
Oct 31 02:14:22.044791 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:22.044748    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"]
Oct 31 02:14:22.094522 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:22.094481    5419 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"]
Oct 31 02:14:22.095768 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:22.095748    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d7915c6-hx5rl"]
Oct 31 02:14:22.096863 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:22.096843    5419 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0392c9d6-h28c2"]
Oct 31 02:14:22.835049 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:22.835016    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2855d516-7749-424b-8bb1-de991fc737cf" path="/var/lib/kubelet/pods/2855d516-7749-424b-8bb1-de991fc737cf/volumes"
Oct 31 02:14:22.835441 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:22.835427    5419 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f7be2b-866d-4e43-8ee7-b9d711600a64" path="/var/lib/kubelet/pods/b0f7be2b-866d-4e43-8ee7-b9d711600a64/volumes"
Oct 31 02:14:23.318401 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:23.318366    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btnvg_af0e6596-aec8-43f1-9aac-2497c9563bef/kube-rbac-proxy/0.log"
Oct 31 02:14:23.844416 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:23.844386    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-btnvg_af0e6596-aec8-43f1-9aac-2497c9563bef/exporter/0.log"
Oct 31 02:14:23.848065 ip-10-0-140-164 kubenswrapper[5419]: I1031 02:14:23.848046    5419 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-l6v7x_eca41a23-4748-4e1d-9527-e635da20d712/discovery/0.log"